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Guidelines  and  discussions  are  presented  for  computer 
perfosaance  evaluation  at  two  levels.  The  first  level, 
Coaputer  Perforaance  Manage  sent  (CPS)  or  Macro  Perfornance 
Evaluation,  involves  an  overall  coaputer  perforaance  marage- 
aent  strategy  concerning  the  use  of  coaputer  resources.  The 
role  of  CPM  throughout  the  coaputer  systea  life-cycle  is 
also  discussed. 

The  second  level  of  coaputer  perfornance  involves 
Coaputer  Perforaance  Evaluation  (CPE)  or  Micro  performance 
Evaluation.  k  brief  discussion  of  CPE  tools  is  given,  as 
well  as  how  to  select  a  perforaance  monitor.  Soae  computer 
perforaance  fallacies  are  revealed  and  a  discussion  of  the 
determination  of  "critical  sections"  of  software  systeas  and 
program  tuning  practices  for  improving  systea  perforaance  is 
presented. 

Limited  discussion  is  devoted  to  performance  issues  in 
relatively  new  areas  in  the  coaputer  field  such  as  networks, 
data  base  aanageaent  systems  and  aicrocoaputers. 
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I.  QMSOim 


&  computer  system  is  a  complex  and  dynamic  mixture  of 
hardware,  software  and  human  resources  that  interact 
together  to  provide  a  service.  Early  computers  had  slow, 
expensive  and  limited  resources.  The  need  to  increase  the 
performance  of  the  system  to  best  take  advantage  of  the 
these  expensive  limited  resources  became  a  challenge  and  a 
necessity.  On  a  batch  system  performance  evaluation  was  far 
less  complicated  then  on  the  multi-task  operating  environ¬ 
ments  of  the  third  generation  computers.  Computer 

Performance  Evaluation  (CPE)  cf  a  computer  system  can  easily 
become  very  expensive  in  many  areas:  system  resources, 

manpower,  and  financial.  Even  after  large  expenditures  in 
all  of  these  areas,  the  results  are  usually  hard  to  inter- 
peret  or  understand  and  may  lead  to  meaningless  conclusions 
that  can  actually  be  of  little  value  in  providing  improved 
performance. 

In  order  to  achieve  successful  and  meaningful  CPE  it  is 
necessary  to  use  Macro  Computer  Performance  Evaluation, 
which  basically  means  to  design,  develop  and  implement  a 
plan  or  framework  to  guide  and  direct  CPE  efforts.  This 
involves  the  coordination  of  both  a  Computer  Performance 
Management  (CPM)  strategy  and  a  technical  evaluation  (meas¬ 
urement  effort)  strategy.  Obviously,  a  CPM  strategy  without 
a  technical  evaluation  strategy  for  gathering  measurements 
will  not  produce  any  useful  results.  On  the  other  hand,  a 
technical  evaluation  or  Micro  Computer  Performance 
Evaluation,  strategy  without  a  well  thought  out  CPM  will 
most  likely  lead  to  the  situation  of  guestionable  or  useless 
results  sentioned  earlier. 


The  objective  of  this  thesis  is  to  provide  a  guide  for 
building  the  foundation  for  both  the  CPU  component  and  th* 
CPE  technical  evaluation  component  for  a  computer  perform¬ 
ance  program.  The  effective  use  of  computer  resources  will 
be  emphasized  and  some  technical  derails  and  expertise 
required  to  carry  out  the  measurement  and  analysis  phase  of 
CPE  will  be  introduced.  Emphasis  will  be  placed  cn  the  idea 
that  it  is  not  necessary  to  get  involved  in  the  more  sophi- 
cated  and  costly  CPE  tools  such  as  software  monitors  and 
hardware  monitors  in  order  to  obtain  noticeable  performance 
enhancements. 

It  is  acknowledged  that  guides  for  CPE  are  not  new  and 
much  research  has  been  co* dieted  producing  abundent  informa¬ 
tion.  However,  the  general  approach  of  past  research  tends 
to  view  performance  issues  from  the  internal  component 
perspective  of  the  computer  system  and  the  emphasis  is  on 
the  efficient  use  of  limited,  expensive  hardware  and  soft¬ 
ware  resources.  This  traditional  view  caused  most  CPE 
efforts  to  rely  upon  expensive  and  sophisticated  hardware 
and  software  monitoring  tools.  Today,  hardware  is  getting 
less  and  less  expensive  and  is  therefore  no  longer  consid¬ 
ered  a  critical  limited  resource.  bore  and  more  computer 
power  is  given  to  the  end  user  and  the  emphasis  of  CPE  is 
switching  from  the  efficient  use  of  the  system  to  effective 
utilization  of  the  system.  This  guide  will  place  more 
emphasis  on  the  effective  use  of  the  system  instead  of  the 
traditionaly  quantatively  oriented,  efficient  use  of  the 
computer  system.  The  hardware  maybe  working  very  effi¬ 
ciently  at  transferring  data  from  secondary  storage  to 
primary  storage,  but  the  user  may  not  have  blocked  the 
records.  fluch  more  effective  use  can  be  made  of  the 
computer  resources  by  blocking  the  file,  therefore 
increasing  the  performance  of  the  computer. 


The  traditional  concept  of  computer  performance  evalua¬ 
tion  aust  broaden  it's  scope  froa  a  aicroscopic  internal 
resource  concentration  to  a  aora  aacroscopic  external 
resource#  total  system  evaluation  CPM/CPE  effort.  People 
and  other  indirectly  coupled  resources  can  have  a  signifi¬ 
cant  impact  on  the  performance  of  the  system.  Many  computer 
facilities  require  tremendous  amounts  of  an  organization's 
resources  to  implement  and  operate.  Performance  of  these 
computer  systems  must  be  managed  to  avoid  critical  perform¬ 
ance  shortcomings  that  jeopardize  the  needs  of  the  organiza¬ 
tion.  Traditional  CPE  tools  and  methods  as  well  as  new  ones 
will  support  the  CPM  effort. 

The  guide  is  intended  for  both  managers  ana  technical 
personnel  who  are  concerned  about  computer  performance 
and/or  are  about  to  become  involved  in  a  CPE  endeavor. 
Managers  will  be  able  to  obtain  a  feel  for  the  scope  and 
depth  of  effort  as  well  as  the  support  required  to  perform 
CPE  effectively.  Inexperienced  technical  people  will  be 
able  to  find  out  what  tools  and  methods  are  available  for 
performing  measurements. 

a  second  objective  of  this  thesis  is  to  explore  the 
impact  new  trends  and  endeavors  in  the  field  will  place  on 
the  current  concept  of  CPE.  It  appears  that  current  CPE 
capability  will  not  be  adequate  for  networks  (individual 
systems  connected  by  communication  links)#  array  processors 
(systems  containing  thousands  of  processors)  #  or  Data  Base 
Management  Systems  (EBHS) .  Ideas  in  some  of  these  areas  are 
presented  in  Chapter  VIII  "Current  and  Future  Technology  and 
Performance''. 

appendix  a#  "Sources  of  Information"#  is  intended  to 
provide  both  managers  and  technical  personnel  with  addi¬ 
tional  sources  of  information  in  their  areas  of  interest, 
appendix  B»  "Examples  of  Performance  Measures"  provides  a 
listing  of  some  of  the  more  common  performance  measurements 
and  arranges  the  measurements  into  five  groups: 
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1.  Quantity  of  work  perforaed 

2.  Quality  of  service 

3.  coeponent  utilization 
Underlying  factors 
workload  characteristics 


4. 

5. 


XX.  INTRODUCTION 


"The  first  computer  had  no  aore  than  two  operational 
pcograas  before  the  idea  occured  to  insert  check  flags  info 
the  coding  to  obtain  a  aeasure  of  how  far  the  program  had 
gotten  before  it.  or  the  machine,  ceased  operation" 
[Ref.  1].  Surprisingly,  aost  manufacturers  of  computer 
systems  as  well  as  managers  and  users  of  these  systems  have 
not  accoaodated  this  interest  in  performance  to  any  appreci¬ 
able  degree.  according  to  Olson  [ Bef .  2]  "Data  processing 
activities  have  been  reluctant  in  the  past  to  address  firm 
perforaance  reguireaents."  This  reluctance  on  the  part  of 
all  personnel  associated  with  the  coaputer  aay  be  attributed 
in  part  to  the  feeling  that  if  no  set  standard  exists,  they 
can  never  be  held  accountable  for  not  aeeting  it.  The 

preliainary  investaent  in  tiae  and  resources  required  by 

• 

aost  coaputer  perforaance  efforts  can  also  be  a  major  factor 
in  postponing  a  serious  interest  in  perforaance  until  a 
crisis  erupts  deaanding  iaaediate  perforaance  attention. 
Soae  siaple  architectural  aodif ications  could  significantly 
aid  the  hardware  aonitoring  effort;  this  is  discussed 
further  in  the  section  on  hardware  monitors.  One  of  the 
oldest  indicators  of  performance  is  the  CPO  busy  (or  CPU 
wait)  light  which  is  usually  located  on  the  front  console  of 
the  coaputer.  However,  the  CPO  busy  light  was  not  intended 
for  perforaance  observation,  but  rather  for  aaintenance 
applications  and  it  is  hard  to  aeasure  the  nuaber  of  micro- 
seconds  of  idleness  by  exaaing  the  CPO  busy  light. 

Horris  also  points  out  that  "the  two  aost  important  and 
effective  tools  for  evaluation  of  coaputer  perforaance  are 
available  free  in  nearly  every  coaputer  installation.  They 
are  y4#ual  inspection  and  gonaop  iiaaj" *  For  exanple,  if 
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the  busy  light  is  constantly  on  (aeaning  CPU  is  busy)  while 
the  prograa  is  executing,  then  the  prograa  could  be  CPU 
bound.  Perhaps  prograa  tuning  could  be  used  to  improve  the 
performance  of  the  prograa.  If  the  CPU  light  is  constantly 
off  and  the  disk  units  are  "dancing  about"  the  computer  room 
floor,  this  indicates  that  the  programs  executing  are  I/O 
intensive,  and  perhaps  the  paging  algorithm  could  be 
adjusted  or  the  blocking  buffer  size  should  be  changed. 
Blocking  buffer  size  is  an  iaportant  performance  issue  and 
is  discussed  in  detail  in  a  later  section.  This  could  also 
indicate  that  there  may  be  very  little  "overlap"  achieved 
between  the  CPU  and  I/O.  Soae  fallacies  that  can  result 
froa  the  use  of  these  tools  are  discussed  in  Chapter  17  "CPE 
Tools  and  the  System  Life  Cycle. 

Another  example,  presented  at  a  computer  performance 
conference,  concerns  the  degraded  performance  of  a  system 
between  the  hours  of  11  A.  H  and  1  P.U.  Shortly  after  the 
installation  of  the  new  version  of  the  operating  system,  the 
System  Administrator  (SA)  began  receiving  coapaints  about 
the  increased  turnaround  time  for  jobs  for  the  period  corre- 
sponding  to  the  two  hour  period  in  which  most  people  took  a 
lunch  break.  The  SA  was  very  puzzled  since  prior  to  the 
installation  of  the  revised  operating  system,  turnaround 
time  on  the  old  operating  system  usually  improved  during 
this  period. 

After  several  days  of  inquiring  about  new  jobs  run,  and 
having  uncovered  no  new  results,  the  SA  called  the  vendor's 
system  analyst  out  of  frustration.  The  SA  mentioned  the 
possible  need  for  a  software  or  hardware  monitor  to  the 
analyst.  The  analyst  suggested  that  they  first  look  at  a 
dump  of  the  accounting  data  for  a  period  shortly  before  and 
after  the  installation  of  the  revised  operation  system. 
Several  new  "system  programs"  began  to  dominate  the  job 
queue  shortly  after  the  installation  of  the  revised  version 
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Figure  2.1  i  Co aeon  Approach  to  Parforaanca  Efforts. 
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CPE  is  usually  first  considered  after  a  system  has  been 
used  for  a  period  of  time  and  becomes  unable  to  provide 
satisfactory  performance  to  either  a  specific  program  cr  tc 
the  whole  system  in  general.  Actually,  CPE  could  be 
performed  at  all  phases  of  the  life  cycle  of  a  computer 
system,  which  typically  includes  the  following  phases: 

•  Procurement  Phase 

•  Installation  Phase 

•  Operation  Phase 

•  Transition  Phase  (tc  a  new  system) 

Due  to  economic  reasons,  formal  life  cycle  CPE  is 
usually  restricted  to  large  mainframes  and  super  computers 
supported  by  large  enough  budgets  to  afford  this  effort. 
Hinicomputer  systems  usually  receive  inexpensive,  very 
informal  CPE  in  the  first  two  phases  where  the  predominate 
goal  is  to  buy  as  much  hardware  (main  memory  and  secondary 
storage-disk  capacity)  as  their  procurement  budget  will 
support  from  the  vendor  who  has  the  best  reputation  for 
quality  system  software  and  software  development  tools. 
This  approach  is  most  likely  motivated  by  the  fact  that  the 
most  expensive  item  in  a  computer  system* s  long  term  budget 
is  software  development  costs,  not  hardware  costs.  However, 
the  software  is  extremely  dependent  upon  the  hardware 
purchased,  and  relies  upon  a  certain  level  of  performance 
from  this  hardware.  If  the  issue  of  hardware  performance  is 
not  adequately  addressed  during  the  procurement  phase,  all 
the  expensive  software  developed  will  be  of  little  value  on 
a  system  that  can't  supply  the  required  level  of 
perforsance. 

Once  the  decision  has  been  sale  to  attempt  CPE  on  a 
system,  a  CPU  program  should  be  initiated,  and  it*s  first 
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task  should  bs  to  establish  a  stratagy  or  define  a  3«t  of 
CPE  objectives  and  their  scopes.  This  is  coverad  in  more 
detail  in  Chapter  III  "Coaputer  Perforaance  ifanageaent" . 

When  atteapting  to  locate  potential  areas  and  bottle¬ 
necks  affecting  systea  perforaance  the  two  aost  iaportant 
and  effective  tools  for  evaluation:  visual  inspection  and 
coaaon  sense  should  be  used.  Host  of  today*s  systeas  have 
accounting  packages  already  installed  and  are  collecting 
valuable  CPE  inforaation,  aainly  about  the  workload.  k 
visual  inspection  of  the  accounting  data  will  give  a  clue  to 
what  kinds  and  types  of  jobs  use  the  aost  resources  on  the 
systea.  Use  of  software  aonitors  and  hardware  nonitors 
should  be  put  off  until  they  are  absolutely  needed  and  can 
be  proved  to  be  cost  effective.  Once  a  hardware  aonitor  is 
introduced  into  the  CPE  effort  the  expertise  and  technical 
resources  reguired  to  support  the  hardware  aonitor  in  order 
to  gain  aeaningful  and  beneficial  results  will  acst 
certainly  becoae  one  of  the  aost  expensive  iteas  in  the  CPE 
budget.  Chapter  IT  CPE  Tools  and  the  Systea  Life  Cycle"  and 
Chapter  V  "Choosing  a  Honitor—  Hardware  or  Software?" 
provides  inforaation  in  this  area. 

Personal  coaputers  (PC* s)  and  networks  are  evolving  and 
gathering  respect  for  their  ability  to  solve  a  wide  range  of 
probleas.  This  is  an  indication  that  decentralized  tech¬ 
nology  and  end  user  coaputer  power  is  on  the  rise.  This 
aeans  that  people  no  longer  have  to  go  to  the  coaputer  in 
the  sense  that  they  aost  leave  their  office  to  obtain  the 
services  of  the  coaputer.  idvancss  in  coeputer  coaauniea- 
tion  technology  has  aade  it  possible  to  take  the  services  of 
the  coaputer  to  the  users  in  their  office.  Decreasing  costs 
as  a  result  of  advances  in  integrated  circuit  technology  has 
eade  it  possible  to  provide  local  coaputer  capability  at 
reasonable  costa.  Perforaance  of  the  "systea"  then, 
concerns  global  as  well  local  considerations. 
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Oar  traditional  views  of  CPE,  as  well  as  oar  nethods,  are 
being  challenged  by  the  energence  of  new  and  expanding 
coaputer  technology  which  deaands  new  aethods  and  approaches 
to  perfora  aeaningful  CPE.  On  the  other  hand,  there  are 
easy  to  use  perf or nance  iaprovsnents,  presented  in  Chapter 
VII  "la proving  Per for nance  Without  donitors",  than  do  not 
receive  wide  spread  use,  bat  coaid  significantly  iaprove  the 
perforaance  of  a  systen.  Chapter  VI  "Sons  Coaaon  Sense 
Perforaance  Fallacies**  reinforces  the  idea  that  hypotheses 
about  perforaance  issues  aust  always  be  validated  by 
aeasureaents. 
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III.  COW  POT  EH  PERFOBHAHCB  3AMSI3MI 


Before  any  professional  carpenter  reaches  for  his  meas¬ 
uring  tape  to  cut  a  peice  of  lumber,  he  will  always  consult 
the  'blue  print'  or  plan  first.  This  is  usually  a  aental 
exercise  in  most  cases,  for  he  has  previously  gone  over  the 
actual  plan  aany  times  in  great  detail,  but  he  also  returns 
to  it  frequently  tc  review  critical  measurements.  The 
important  issue  is  that  he  works  from  a  plan.  This  saves 
time,  coordinates  his  activities  toward  a  predetermined  goal 
and  avoids  waste.  The  above  approach  should  also  be  rele¬ 
vant  for  anyone  involved  in  CPE. 

In  FIPS  PUB  49  [Bef.  4  ],  a  CPM  program  is  defined  as 
"any  .structured  effort,  in  house  or  otherwise,  to  measure 
and  evaluate  the  performance  of  a  computer  facility  in 
support  of  established  management  goals  and  objectives" .  In 
addition,  the  CPH  program  should  strive  to  maintain  the 
computer  system  at  the  highest  perforaance-to-cost  ratio  and 
provide  management  with  reports  on  the  status  of  the 
computer  system.  These  reports  should  be  simple,  short,  and 
easily  understandable  by  management. 

The  immediate  objective  of  a  CPH  strategy  is  not  to 
troubleshoot  the  system  or  to  do  system  tuning,  but  rather 
to  maintain,  or  in  some  instances,  regain  control  of  a 
complex,  costly  and  critical  resource  through  a  quantitative 
and  qualitative  understanding  of  how  that  resource  performs 
and  of  the  alternatives  that  ara  avaliable  to  make  it 
perform  more  effectively  and  efficiently.  Thus,  CPH  is 
obligated  to  serve  management,  who  initially  approved 
funding  for  the  computer  facility  resource,  and  to  the  users 
who  must  use  the  systea  to  meet  the  organization's  estab¬ 
lished  goals  and  objectives. 
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1.  GBTTIMG  STABTBD — THE  FIBST  STEP 

Management  a  us*  realize  that  computers  directly  affect 
an  organization's  bottom  line.  This  effect  is  expected  to 
increase  substantially  in  the  future,  due  to  the  unusual 
nature  ox  the  computer  products  market  where  physical  size 
of  the  system  is  decreasing,  capacity  and  capability  are 
increasing  and  prices  are  decreasing.  Thus,  the  mainframe 
computer  of  yesterday,  usually  found  in  a  computer  room  away 
from  people,  is  now  showing  up  in  small  packages  on  the  tops 
of  desks  in  the  office  enviornment. 

Spinelli  [  Bef.  5]  defines  the  role  and  responsibilities 
of  the  information- systems  manager  who  will  most  likely  rely 
upon  the  CPU  group  to  provide  most  of  the  information  he 
needs.  Therefore,  the  CPU  manager  must  be  aware  of  this 
role  and  be  capable  of  supporting  the  following 
responsibilities: 

1.  Enable  the  organization  to  understand,  apply  and 
ccntrol  internal  forces,  and  properly  utilize 
external  forces,  that  shape  a  firm's  computing  envi¬ 
ronment. 

2.  Apply  proven  management  principles,  particularly 
strategic  planning,  to  information-systems  functions. 

3.  Provide  the  organization  with  planning  and  develop¬ 
ment  concepts  and  tools  that  effectively  enable  the 
firm  to  develop  and  implement  a  corporate  computing/ 
business  strategy,  and  to  provide  a  logical  framework 
in  which  it  can  understand  newest  usage  trends  in 
information  systems  management  and  computer  tech¬ 
nology. 

4.  Provide  the  organization  with  a  forum  to  exchange 
ideas  and  discuss  opportunities  with  information 
systems  specialists. 
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5.  Comprehend  the  critical  nature  of  information  systems 
functions  in  the  context  of  overall  organizational 
success. 

k  wide  range  of  computer  performance  measuraents  are  the 
primary  source  of  information  for  the  above  areas. 

In  the  aggregate,  computer  systems  represent  the 
totality  of  the  organization  itself,  in  that,  cne  vital 
corporate  resourca--infornation--is  provided  to  the  other 
vital  corporate  resource— people.  Therefore,  timely  rele¬ 
vant,  accurate  information,  effectively  produced  and  disse¬ 
minated,  is  a  vital  corporate  resource.  In  this  light,  c?M 
must  use  CPS  in  providing  this  resource  in  the  most  effi¬ 
cient  and  effective  manner. 

Regrettably,  one  external  reason  for  the  existence  of  a 
CPU  group  in  the  Pederal  Government  and  the  DoD  is  the 
extremely  long  procurement  time  required  to  order  new  equip¬ 
ment.  Typically,  if  the  workload  on  a  system  becomes  too 
much  and  extensive  enhancements  or  a  new  system  is  needed, 
it  will  take  a  minimum  of  one  year  to  obtain  the  desired 
equipment  using  the  extremely  long  and  paperwork  intensive 
HOP  procurement  process  required  by  the  Federal  and  DoD. 

On  the  positive  side,  steps  must  be  taken  to  maintain  the 
best  system  performance  during  this  procurement  time,  which 
motivates  the  organization  to  implement  CPH  and  CPE. 

i.  saia&in  lan  flatten 

In  order  to  develop  a  CPB  plan  it  is  necessary  to 
select  personnel  to  form  a  CPH  team  or  group.  It»s  politi¬ 
cally  wise  to  select  at  least  one  represenative  from  each  of 
the  various  departments  that  can  directly  influence  the 
performance  of  the  computer  system  in  order  to  insure 
overall  cooperation.  It  is  both  politically  wise  and 
economically  essential  to  include  a  representative  from 
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upper  management,  since  they  traditionally  are  responsible 
for  allocating  funds  for  such  groups  and  can  give  the  CPS 
group  the  needed  recognition  and  importance  to  sustain  an 
effective  role  in  the  organization. 

Each  representative  need  not  be  a  full  time  active 
■eeber  of  the  group,  but  should  remain  aware  of  the  main 
goal  of  the  CPU  effort.  Pull  time  participation  of  all 
members  is  not  very  practical  in  a  large  organization  with 
many  departments,  since  it  would  tend  to  make  the  CPM  group 
too  large  and  possibly  ineffective.  Small  organizations 
could  have  difficulty  also,  since  it  may  not  be  possible  to 
dedicate  many  people  to  this  effort  on  a  full  time  basis  due 
to  lack  of  personnel.  If  possible,  a  minimum  ’core'  of 
fulltime  CPM  personnel  should  be  choosen,  that  are  well 
insulated  from  "fire  fighting"  and  the  day  to  day  disjoint 
crisis  situations  which  usually  exist  in  a  computer  system 
work  environment  to  insure  continuity  in  the  CPE  effort. 

If  the  organization  forming  a  CPM  group  is  part  of 
the  Federal  Government  or  the  Department  of  Defense  (DoD) , 
two  organizations  can  provide  a  great  deal  of  help  since 
both  organizations  are  dedicated  to  helping  in  the  area  of 
CPS  and  CPE.  The  Federal  Government  organization  is  known 
as  P  EDS  IE  (Federal  Automatic  Data  Processing  Simulation 
Center).  See  [Ref.  6]  for  location  and  contact  information. 
Quoting  from  the  organization's  brochure:  "FEDSIM  provides  a 
central  source  for  computer  simulation  services  so  that 
individual  agencies  will  not  need  to  develop  independent 
capabilities  to  use  advance  techniques  of  computer  perform* 
ance  measurement  and  evaluation.  This  will  allow  all  agen- 
cies  to  have  access  to  powerful  techniques  on  a  cost 
reimbursement  basis  without  incurring  high  individual 
start-up  costs  in  time,  money  and  expertise". 

FEDSIM  policies  are  established  by  a  Joint  Policy 
Committee  made  up  of  representatives  of  gs A,  Air  Force,  the 
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Office  of  the  Secretary  of  Defense  and  the  National  Bureau 
of  Standards.  FZDSIM  provides  the  following  services,  tools 
and  techniques: 

1 .  SERVICES 

a)  Computer  performance  evaluation  consultant 
services  and  technical  assistance  for: 

i)  Systeas  design 

ii)  Systems  specification 

iii)  Coaputer  equipment  configuration 

iv)  Coaputer  program  improvement 

v)  ADP  equipment  selection 

b)  Contractual  assistance  for  purchase,  lease  or  use 
of: 

i)  Simulation  packages  and  languages 

ii)  Hardware  monitors 

iii)  Software  monitors 

iv)  Analytical  techniques 

c)  Training  in  the  application  of  coaputer  perform¬ 
ance  measurement  and  evaluation  techniques 

2.  TOOLS  AND  TECHNIQUES 

a)  Simulation  Packages 

i)  COMET,  Coaputer  Operated  Machine  Evaluation 
Technique  (commercially  available  as  SCERT, 
Systems  and  Computer  Evaluation  and  Review 
Technique) 

ii)  CASE,  Computer  Assisted  System  Evaluation 

iii)  SAM,  Systeas  Analysis  Machine 

b)  Simulation  Languages 

i)  GPSS ,  General  Purpose  System  Simulation 

ii)  SIMSCRIPT 

iii)  GASP,  General  Activity  Simulation  Program 

c)  Hardware  Monitors 

d)  software  Monitors 


a)  Analytical  Routines 

FEDSIM  was  established  by  tha  GS A  and  is  operated  by 
the  Air  Force  at  the  request  of  gsa. 

The  second  organization  is  the  Navy  Regional  Data 
Automation  Center  (NARDAC),  Pensacola.  See  [Ref.  7]  for  the 
complete  organization  title  and  location.  This  organization 
has  the  responsibility  to  provide  tools,  methods,  and  proce¬ 
dures  for  Computer  Performance  Evaluation/Capacity 
Management,  as  well  as  the  resposibility  to  provide  training 
and  consultation  services  to  Navy  activities  as  required. 

According  to  S.  B.  Olson  [Ref.  2]  "The  first  fact  to 
come  to  light  was  that  research  indicated  that  effective 
Performance  Management  functions  were  simply  not  being 
accomplished  within  Navy  data  processing  activities.  In 
fact,  evidence  suggested  that  throughout  industry  as  well, 
there  was  relatively  little  well  organized  and  systematic 
Performance  Evaluation  or  Capacity  Planning  being  done." 

One  of  the  primary  reasons  perceived  by  Olson  as 
hampering  good  performance  and  capacity  management  was  the 
"general  lack  of  a  stand-alone  specialized  group  whose 
primary  responsibility  was  the  performance  management 
effort".  Typically,  it  was  found,  when  studies  were  made  or 
data  collected,  the  effort  was  usually  assigned,  perhaps  by 
default,  to  a  person  or  group  of  day-to-day  production 
people  at  the  data  processing  center. 

Obviously,  as  pointed  out  earlier,  in  a  small  group, 
this  problem  would  be  hard  to  eliminate  without  a  noticable 
loss  of  support  in  other  areas.  However,  it  does  indicate 
that  for  larger  groups,  management  was  either  not  involved 
or  involved  but  unaware  of  the  potantial  benefits  of  a  well 
organized  CPU  program  and  therefore  treated  the  CPM  efforts 
with  low  or  no  priority  when  "fires"  occured.  One  possible 
solution  to  this  problem  is  to  educate  management  as  to  the 
potantial  benefits  of  a  CPM  program  and  assign  a  priority  to 
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CPU  that  is  high  enough  so  that  it's  objectives  can  be 
accomplished  uninterrupted.  Again,  it  is  suggested  that  at 
least  one  person,  possibly  rotating  the  role  aacr.gst 
different  members  in  the  group,  be  iad9  the  continuing  full¬ 
time  active  member  in  the  CPS  group.  This  should  ensure 
that  the  CPU  effort  does  not  stall,  but  continues  tc  meet 
its  objectives. 

2.  naiaiaa  Iks  ££&  isai 

Once  selected,  the  CPH  team  oust  become  veil 
acquainted  with  many  ill  defined  areas  concerning  CPM  and 
CPB  such  as:  present  system  workload,  projected  system  work¬ 
load,  current  computer  capacity,  and  most  importantly  the 
objectives  and  the  goals  of  the  organization  and  how  the 
computer  center  will  aid  in  meeting  them.  A  first  step  for 
management  training  for  any  CPB  group  which  is  a  part  of  the 
Federal  Government  would  be  to  contact  either  FEDSIM  or 
NABDAC.  Potential  management  training  for  non  Federal  or 
DoD  CPM  groups  would  have  to  rely  on  firms  in  private 
industry  that  provide  consulting  services  in  the  CPM  and  CPE 
fields.  One  such  firm,  founded  by  M.  F.  Morris  coauthor  of 
[Bef.  1].  is  Management  Advisory  Service,  located  in 
Kashington  DC.  Mr.  Morris  was  with  FSDSIM  in  the  past.  A 
government  agency  that  does  offer  printed  information 
relating  to  CPH  and  CPE  is  the  National  Bureau  of  Standards, 
see  [Bef.  8]. 

One  specific  source  that  provides  extensive  informa¬ 
tion  on  capacity  planning  is  "Capacity  Planning:  1  State  of 
the  Art  Survey",  [Ref.  9].  This  reference  presents  a  very 
detailed  discussion  of  capacity  planning  and  provides  many 
recent  references.  Other  sources  of  information,  including 
CPE/CPH  users  groups  and  conferences,  which  usually  have 
published  proceedings,  can  be  found  in  the  Appendix  listing 
of  sources  of  information. 
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B.  CPH  BE  S POM S I BI  LI  TIBS 


According  to  PIPS  PUB  49  [Ref.  4]  CPU  responsibilities 
will  deal  with  at  least  four  major  areas: 

•  User  Requirements 

•  System  Resource  Management 

•  Communicating  Bith  Upper  Management 

•  Vender  Relations 

Prior  to  beginning  a  discussion  in  the  above  areas  it  must 
be  pointed  out  that  these  areas  can  help  an  organization 
meet  both  its  short  and  long  range  goals  and  objectives. 
The  CPH  group  should  obtain  a  written  description  or  state¬ 
ment  of  management's  ideas  and  expectations  of  how  the 
computer  system  and  its  related  resources  are  responsible 
for  helping  to  meet  the  organizations  long-term  and  short¬ 
term  goals  and  objectives.  This  may  require  repeated 
discussions  in  order  to  fully  understand  exactly  what 
management  means  and  expects.  In  many  cases  management  is 
most  likely  poorly  informed  about  how  well  the  computer 
facility  is  actually  meeting  the  organizations  goals  and 
objectives  or  what  the  possible  effect  of  increasing  the 
current  workload  will  have  on  the  performance  of  the  system. 
It  is  important  that  management  is  informed  about  it's 
computer  facility.  It  is  also  important  that  the  CPH  team 
continue  this  dialogue  with  management  in  order  to  be  aware 
of  future  developments  that  may  have  impact  on  the  immediate 
or  future  performance  of  the  computer  system.  A  well 
informed  management*  especially  in  the  area  of  how  well  the 
computer  system  is  performing*  will  be  in  a  much  better 
position  to  understand  the  computer  facilities  problems  and 
needs*  and  will  be  more  apt  to  take  a  positive  role  in 
procuring  recommended  resources  or  change.  An  awareness  of 
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management's  near-term  and  future  plans  will  aost  likely 
influence  the  CPU  teaas  r ecoamendations  to  solve  current 
perforaance  probleas  based  upon  current  workloads. 
Procurement  of  equipment  to  provide  adequate  perforaance  in 
terms  of  today’s  workload  may  not  be  sufficient  to  provide 
acceptable  perforaance  in  the  near  future. 

1  •  22££  Maims 

Osers  traditionally  view  the  computer  system  from  a 
very  high  level  in  terms  of  perforaance.  They  are  generally 
interested  in  turnaround  time  or  response  time*  accessi¬ 
bility,  reliability  and  availability  of  the  system.  If  poor 
perforaance  exists  in  these  areas  the  user  can  become  frus¬ 
trated  and  the  "performance"  of  rhe  user  or  user  resource 
can  be  adversely  effected.  If  allowed  to  exist,  this  situ¬ 
ation  can  cause  conterproductive  tension  between  the  user 
and  the  computer  system  where  the  system  includes  the  group 
of  people  who  run  and  maintain  the  system. 

&  negative  attitude  can  displace  an  enviornaect  of 
constructive  comments  with  one  of  unconstruct ive  fault 
finding  criticisms  that  can  easily  turn  into  a  cynical  view 
of  the  computer  system  by  the  users.  If  allowed  to 
continue,  this  situation  can  evolve  into  a  situation  of 
conscious  or  unconscious  acts  of  sabatoge  on  both  the  users 
part  and  on  the  computer  management  group.  The  obvious  net 
result  in  this  type  of  situation  is  a  gross  reduction  in 
overall  system  perforaance  produced  predominately  by 
external  forces  (user  attitude)  that  may  have  been  influ¬ 
enced  by  some  internal  ones  (slow  response  time)  . 

a.  Timeliness 

Few  users  actually  keep  track  of  the  exact  tines 
for  execution  of  their  jobs,  but  rather  develop  an  intuitive 
feel  for  these  times.  The  CPH  team  should  therefore  obtain 
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these  tiaes  from  systea  accounting  log  files.  This  source 
of  infcraation  about  batch  jobs  works  quite  well.  However, 
for  interactive  jobs  that  aeasura  perforaance  through 
response  tiae  other  aethods  aust  used  .  The  development  of 
reaote  terainal  eaulators  (BTE's)  and  intelligent  hardware 
aonitors  has  aade  it  possible  to  accurately  measure  interac¬ 
tive  response  tiae  of  a  large  nuabec  of  terainals.  The  RTE 
concept  uses  a  separate  coaputer  froa  the  one  which  is  being 
aeasured  to  serve  as  a  terainal  activity  generator  (RTE) . 
The  RTE  generates  activity  that  looks  to  the  system  being 
measured  as  if  it  was  the  activity  of  some  ' r.  •  terainals. 
"Usually,  the  programs  in  the  RTE  are  parameter-driver,  and 
aay  be  altered  over  a  vide  range  of  emulated  activity", 
(Ref.  1].  One  very  siaple  method  used  to  obtain  a  relative 
measure  for  response  tiae  is  to  keep  track  of  the  response 
tiae  of  a  aiz  of  prograas,  that  remain  unaltered  or  'fixed' 
and  are  executed  at  the  sane  tines  throughout  a  workload 
period  (hour,  day,  week,  etc.) .  The  six  should  include  jobs 
that  are  I/O  intensive  and  CPU  intensive.  Froa  the  CPH 
point  of  view,  RTE-like  monitors  can  provide  an  accurate 
record  of  users  interactions,  which  reveal  exactly  what 
level  of  service  each  user  is  actually  receiving. 

Jobs  should  be  given  a  priority  and  classified 
according  to  their  use  of  resources.  For  example,  to  take 
advantage  of  CPU  and  I/O  overlap,  experiments  3hould  be 
conducted  to  find  which  jobs  in  the  saae  priority  class  are 
CPU  intensive  and  which  jobs  are  1/3  intensive.  In  order  to 
achieve  sore  effective  perforaance  of  the  systea  jobs  with 
CPU  intense  operations  should  be  scheduled  with  jobs  having 
intensive  I/O  operations.  Obviously,  two  jobs  competing  for 
the  saae  resource  (either  I/O  or  the  CPU)  will  experience 
longer  turnaround  tiae. 

Since  perforaance  can  be  severely  reduced,  it 
can  be  counter  productive  to  schedule  debug  and  developaent 
tiae  during  tiaes  of  intense  production  runs. 
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Timeliness  can  be  a  function  of  output  aeii.ua 
and  computer  systea  adainistration  policy.  This  comes  down 
to  placing  a  value  cc  resources:  user  resource  (time) 
versus  systea  resources  (printer  paper,  cpu  tine).  For 
example,  at  The  Naval  Post  Graduate  School,  upper  and  lover 
case  printer  output  for  students  for  a  specific  job  is  only 
available  once  every  twenty-four  hours.  Thus,  the  effective 
turnaround  tiae  for  jobs  reguiring  upper  and  lover  cass  text 
is  twenty-four  hours!  This  becomes  very  interesting  at  the 
end  of  an  academic  session,  especially  when  graduating 
students  are  trying  to  finish  thesas.  It  is  possible  that 
adequate  student  training  aimed  at  reducing  the  aaount  of 
printer  output,  could  increase  the  system’s  overall  effec¬ 
tive  performance. 

Response  and  turnaround  tine  is  also  a  function 
of  coaaunication  mediua.  Modems  are  a  good  example.  A 
modes  capable  of  300  baud  is  noticiably  slower  than  one 
running  at  1200  baud.  If  it  takes  fifteen  minutes  to 
transfer  a  file  on  a  1200  baud  modem,  it  will  take  one  hour 
to  transfer  that  same  file  on  a  303  baud  modem.  This  is  a 
four  hundred  percent  decrease  in  overall  system  performance. 
In  this  case,  the  user  could  find  something  else  to  do  while 
waiting  for  the  transfer  to  complete.  However,  in  an  inter¬ 
active  situation  taking  15  minutes  to  complete  using  a  1200 
baud  modem  could  take  as  long  as  one  hour.  This  is  also 
true  with  networks  and  the  coaaunication  medium  when  they 
must  use  public  communication  lines  to  communicate  form  node 
to  node.  Thus,  the  performance  of  a  node  reguiring  service 
from  the  network  can  be  greatly  influenced  by  the  communica¬ 
tions  link,  an  external  resource  to  the  node's  computer 
systea. 

Response  tiae  of  a  user  can  effect  the  perform¬ 
ance  of  the  system  since  the  response  of  the  computer  is 
dependent  upon  the  user  giving  some  command  or  answering  a 
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query  in  an  interactive  situation.  The  response  rime  of  a 
user  who  aust  depress  several  keys  to  enter  a  string  of 
alphanuaeric  characters  to  enter  a  coaaand  or  respond  tc  a 
query  will  be  much  greater  than  a  user  who  aust  siaply 
depress  a  function  key,  uses  a  light  pen,  or  a  touch  panel. 

This  is  especially  true  and  aost  frustrating  to  a  user  who 
has  little  or  no  experience  at  a  keybord.  For  highly  inter¬ 
active  sessions,  the  "souse"  (an  input  device  that  resides 
on  any  flat  surface  next  to  the  tarainal  screen  and  whose 
wove  sent  about  the  flat  surface  is  refected  in  the  aoveaent 
of  the  cursor  on  the  terainal  screen)  provides  a  much  aore 
desirable  huaan-aachine  interface  resulting  in  iaporved  user 
respose  tiae  and  overall  increased  perforaance.  The  user  of 
a  aouse  siaply  positions  the  cursor  over  the  desired  choice 
of  response  displayed  on  the  screen  and  depresses  a  button 
located  on  the  aouse. 

b.  Accessibility 

Basically  this  involves  being  able  to  convi- 
nently  get  at  all  ccaputer  systea  resources  necessary  to 
coaplete  a  desired  task.  If  a  person,  who  constantly  uses 
the  coaputer  systea,  aust  leave  their  office  area  to  gain 
access  to  a  terainal,  then  the  terainal  is  not  considered 
very  accessible.  These  resources  include  copies  of  systea 
and  resource  docuaentation  (to  include  terainal  sign  on/off 
proceedures,  printer/plotter  operating  instructions,  etc.) 
as  well  as  hardware  resources. 

Systea  resource  deaand  and  requireaent  statis¬ 
tics  can  be  obtained  froa  systea  accounting  log  files.  Froa 
this  inforaation  the  aost  effective  geographical  placeaent 
of  resources  can  be  deterained  that  will  provide  the 
greatest  benefit  to  the  users  as  a  group. 


c.  Reliability 


Frequent  system  crashes  or  down  time  can  be  a 
nuisance,  especially  to  the  interactive  users.  information 
should  be  recorded  on  mean- time-between-failure  (dTBTF)  and 
eean-time-to-repair  (MTTR)  .  &  high  frequency  of  system 

failures  could  indicate  a  weak  hardware  or  software  link  or 
bug  in  the  system.  Problems  in  this  area  are  very  hard  to 
solve  due  to  inadequate  information.  Again,  accounting  log 
files  could  help  uncover  a  pattern,  possibly  related  to  a 
particular  job  or  combination  of  jobs  running  concurrently 
that  could  be  related  to  the  system  problem.  More  discus¬ 
sion  on  this  subject  is  in  the  section  on  Vendor  relations. 

d.  Availability 

In  a  very  broad  sense  availability  is  the 
percent  of  the  total  time  the  system  can  directly  serve  the 
users  to  help  meet  the  goals  and  objectives  of  the  organi¬ 
zation.  Scheduling,  system  maintenance,  system  failures  and 
repairs  influence  system  availability.  An  extreme  example 
dealing  with  both  availability  and  accessibility  concerns 
the  solution  of  one  computer  facility  to  a  problem  of  insuf¬ 
ficient  terminal  availability.  There  were  simply  not  enough 
terminals  for  the  demand  of  users  who  needed  then.  The 
"solution"  devised  by  the  system  administrator  was  to  allow 
users  to  log  on  only  once  each  day.  Any  attempt  to  log  on 
the  system  again,  in  the  same  day  by  the  same  user,  would  be 
denied.  This  "solution"  caused  people  to  "hang  on"  to  their 
terminal  by  running  endless  jobs  while  they  were  not  actu¬ 
ally  using  the  terminal,  so  that  they  could  have  it  avail¬ 
able,  if  needed  later  in  the  day.  A  better  solution  may 
have  been  to  extend  the  shift  and  schedule  users  for 
specific  terminals. 


2.  o§££  samii 


Support  functions  such  as  training  seminars,  user 
consultation,  system  documentation  and  the  need  for  regular 
computer  center  news  bulletins  are  difficult  requirements  to 
determine  and  evaluate. 

Once  again,  the  availability  of  detailed  accounting 
data  can  be  of  use  in  this  area.  Frequency  or  infrequency 
of  use  by  users  of  system  software  tools  and  system 
resources  in  general,  as  well  as  frequency  of  compiles  can 
help  determine  which  users  may  require  initial  training  or 
additional  training  in  the  use  of  system  resources.  A  new 
programmer  with  a  very  optimistic  outlook  always  included 
the  catalog  job  control  statement  after  the  compile  state¬ 
ment  in  all  software  under  development.  Consequently,  the 
operating  system  would  catalog  a  job  even  though  the 
compiler  encoutered  fatal  errors  and  the  compile  was  unsuc¬ 
cessful.  The  programmer  was  needlessly  wasting  system 
resources  and  contributing  to  inefficient  use '  of  the 
computer.  Apparently,  the  programmer  was  unaware  that  the 
operating  system  did  provide  for  a  conditional  job  control 
statement  which  would  have  prevented  the  catalog  if  the 
compile  were  unsuccessful.  Or,  possibly  the  programmer 
added  or  deleted  some  code  to  an  existing  program  and  failed 
to  take  out  the  catalog  statement  until  assured  that  the 
compile  was  successful.  If  a  new  software  tool  is  infre¬ 
quently  used,  a  training  session  nay  be  required. 

It  is  also  important  to  maintain  a  well  run  and 
responsive  user  problem  report  or  suggestion/comment  system. 
Osers  are  a  very  good  source  (sometimes  the  only  source)  for 
finding  system  hardware  and  software  problems  or  bugs. 
Osers  can  be  encouraged  to  participate  in  the  system  by 
always  providing  quick  and  immediate  sincere  feedback,  if 
only  to  reveal  the  status  of  their  report  or  comment,  and 
thank  then. 


W*  ifemr-. 


3.  2§5£  sasaatss  glUisuiaa  *12.211 

Many  computer  facilities  must  use  accounting  data 
for  user  charge  back  systems  which  are  necessary  to  obtain 
the  funds  required  to  operate  and  maintain  the  computer 
facility.  These  reports  are  usually  very  high  level,  gener¬ 
ally  providing  totals  for  categories  of  resource  usage,  but 
the  system  uses  more  detailed  information  to  generate  these 
totals.  This  detailed  information  about  user  resource 
utilization  should  be  made  into  a  User  Resource  Utilization 
Report  and  given  to  each  user  showing  by  job,  what  resource 
is  being  used  and  at  what  amount.  The  user  should  be 
adequately  trained  to  take  advantage  of  this  information. 
Once  accountability  for  computer  resource  utilization  is 
established  and  made  known  to  the  users,  they  are  frequently 
encouraged  to  cut  down  on  wasteful  use  of  computer 
resources.  This  can  be  especially  true  in  the  case  when  one 
user  who  needs  a  resource  or  more  of  a  resource,  knows  what 
user  or  users  are  possibly  wasting  that  resource  or  using  it 
inefficiently.  The  net  effect  can  be  a  reduction  of  user 
demand  for  critical  resources  and  an  increase  in  system 
performance. 

It  should  be  mentioned  that  accounting  packages  span 
the  range  of  levels  of  detailed  information  they  collect 
about  jobs  in  the  system  and  about  the  system. 
Traditionally,  larger,  more  expensive  systems,  like  I  BN, 
have  accounting  systems  whose  log  files  contained  more 
detailed  information  about  individual  jobs  (number  of  I/O 
requests,  amount  of  memory  used,  number  of  pages  of  printed 
output,  etc.)  and  the  system;  whereas  smaller,  less  expen¬ 
sive  systems  have  accounting  data  of  less  detail.  In  few 
cases  is  it  found  that  the  accounting  log  files  contain  all 
the  information  required,  and  supplemental  software  is 
required  to  gather  this  additional  information.  Reviewing 
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the  source  code  listing  and  the  documentation  of  an 
accounting  prograa  can  proivide  a  great  deal  of  ir.f  ora  anion 
about  what,  where  and  how  to  gather  data  on  job  and  syscaa 
performance. 

C.  BESOOBCE  BAB&GEHEHT 

The  CPH  group  should  be  able  to  easily  monitor  which 
resources  are  underutilized  (for  example  cardreaders,  7 
track  tape  units,  etc.)  and  which  resources  are  hawing 
difficulity  meeting  the  workload  demands.  Using  this  infor¬ 
mation,  obtainable  through  the  accounting  package  and  modi¬ 
fications  to  it,  the  CPM  group  will  be  able  to  make  a 
report,  reguarding  elimination  of  under  utilized  resources 
(shifts,  equipment,  etc.)  or  the  procurement  of  additional 
resources  (memory,  disk  drives,  tape  drives,  etc.).  Through 
these  computer  performance  evaluation  efforts,  direct  cost 
savings  as  well  as  improved  performance  of  the  system  can  be 
realized  by  the  computer  facility. 

Information  collected  in  a  historical  database  in  this 
area  can  be  an  invaluable  aid  in  predicting  the  near  and 
long  term  pattern  of  growth  expected  for  the  computer 
facility.  "Number  of  jobs  completed  per  month,  percent 
utilization  of  major  system  resources,  hours  of  system 
availability  are  several  measures  applicable  to  this 
problem.  Performance  data  is  therefore  valuable  not  only 
for  enhancing  the  present  system,  but  also  for  constructing 
models  of  future  resource  requirements'*.  (Bef.  4] 

i .  soiasaisatiaa  silk  g&en  saaaaaisat 

It  should  be  a  prime  responsibility  of  the  computer 
facility  to  report  to  upper  management  on  the  status  of  how 
well  the  computer  facility  is  meeting  it's  part  in 
supporting  the  organizations  objectives  and  goals.  Computer 
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system  performance  is  a  key  issue  in  this  commitment  and  the 
CPS  group  can  provide  auch  of  the  required  inforaatior.  and 
recoaendations  in  this  area.  Reports  should  be  utilized  to 
convey  this  information  to  both  the  computer  facility  ar.d  to 
upper  aanageaent.  In  addition  to  these  reports  the  computer 
facility  manager  may  require  additional  information  about 
the  performance  of  the  system  at  a  more  detailed  ?evel. 

The  form  and  format  of  these  reports  can  vary 
greatly  due  to  each  organizations  reporting  requirements, 
but  some  general  guidance  can  be  given  CRef.  1],  [Bef.  4]  : 

•  Status  reports  should  be  regular,  concise,  and  prefer¬ 
ably  graphical  in  nature. 

•  The  amount  of  information  reported  should  not  exceed 
management  requirements.  "Too  auch,  too  often"  is  a 
problem  common  to  many  performance  reporting  schemes. 

•  Information  should  be  at  a  level  of  abstraction  which 
upper  management  can  easily  digest  and  understand,  but 
sufficient  to  support  the  decision  making  process. 

•  The  reports  should  compare  the  center's  current  level  of 
performance  against  a  set  of  predefined  goals. 

2 .  Vendor  Relations 

Host  computer  facilities  interface  regularly  with 
vendor  marketing  and  technical  support  personnel.  A  positive 
relationship  with  technical  personnel  can  be  invaluable 
especially  in  the  areas  of  hardware  and  software  reliability 
and  maintenance.  k  joint  review  of  hardware  and  software 
performance  data  on  a  regular  scheduled  basis  by  the 
computer  facility  personnel  and  vendor  representatives  will 
help  foster  mutual  respect  and  understanding.  Tracking 
system  problems  such  as  the  frequency  of  tape  and  disk 
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parity  errors,  the  cause  and  duration  of  system  cr  as  has ,  and 
variations  in  other  system  performance  measures  will  provide 
facility  manager  with  a  folder  of  factual  information  to 
document  the  occurrence  of  these  problems  with  which  the 
vendor  must  deal.  The  capability  to  identify  the  source  as 
well  as  the  existence  of  errors  becomes  especially  important 
in  the  multi-vendor  computer  system  enviornaent,  in  which 
each  vendor  points  at  the  other  vendor  as  being  the  main 
source  of  the  problem. 

Any  proposed  hardware  and/or  software  modification 
will  most  certainly  affect  the  performance  of  the  system. 
They  must  therefore  be  evaluated  from  the  total  system  point 
of  view,  not  from  just  one  aspect,  as  to: 

•  Utility 

•  Cost  Ef feet iveness 

•  Impact  cn  total  System  Performance 

• 

"4  modification  to  a  system  scheduler  which  is  intended  to 
increase  batch  throughput  but  which  inadvertently  degrades 
interactive  response  time  fails  to  consider  total  system 
performance.  Measurement  tools  and  techniques  can  be  used 
not  only  to  detect  performance  problems,  but  also  to  antici¬ 
pate  them  and  to  prevent  their  occurrence."  [Ref.  *] 

D.  IISTITUTIHG  THE  CPE  PROG  BIB 

Measurement  and  evaluation  techniques  are  available  to 
support  the  efficient  and  effective  management  of  a  computer 
facility.  How  to  introduce  them  into  the  computer  system 
enviornment  can  be  a  problem.  Some  questions  from  [Ref.  4], 
properly  asked  are: 
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•  How  often  should  the  inforaation  obtained  from  perform¬ 
ance  data  be  reported  to  the  computer  facility  adminis¬ 
trator? 

•  In  what  form  3hould  it  be  reported? 

•  Hhat  is  the  role  of  the  computer  facility  administrator 
in  instituting  CPM  procedures? 

1.  Heportincr 

Figure  3.1  [  Bef .  4],  shows  typical  computer  system 
life-cycle,  progressing  from  an  analysis  of  requirements  to 
the  final  installation,  operation,  and  enhancement  of  the 
selected  system.  In  each  phase  of  the  computer  life-cycle 
performance  measurement  and  evaluation  should  provide  a  base 
for  satisfying  the  informational  needs  of  the  computer 
facility  administrator.  Performance  data  is  as  useful 
during  the  requirements  analysis  phase  as  it  is  during  the 
system  enhancement  phase.  Every  installation,  reguardless 
of  size,  should  incorporate  a  system  performance  reporting 
strategy  during  each  phase  of  its  system's  life-cycle. 

Availability  alone  should  not  be  used  to  determine 
the  types  of  data  to  be  collected  and  reported.  Instead, 
the  data  should  first  be  selected  as  to  the  informational 
requirements  of  the  computer  facility,  which  are  determined 
by  the  computer  facility  administrator's  scope  of  responsi¬ 
bility.  Each  report  should  provide  a  historical  trend  of 
the  facility's  performance.  Updates  should  be  performed  on 
a  regular  basis  (depending  upon  the  nature  and  importance  of 
the  inforaation) ,  and  should  contain  specified  performance 
criteria.  Control  limits  should  be  established  from  this 
criteria  and  placed  on  the  performance  charts.  Control 
limits  are  a  values  chosen  to  represent  the  boundaries  of 
acceptable  performance  for  a  given  system  variables.  These 
variables  and  their  associated  control  limits  could  be 
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objective- direct ed  and  indicate  the  facility's  ability  to 
meet  certain  specified  objectives  such  as  one-hour  average 
batch  turnaround  time  and  3  second  terminal  response  time. 
Others  are  process-directed  indicating  the  level  of  perform¬ 
ance  of  internal  system  resources  like  the  CPU,  disks,  ana 
memory.  Exception  reports  should  be  generated  whenever 
control  limits  are  exceeded.  When  appropriate,  an  in-depth 
study  may  be  recommended  to  determine  the  specific  cause  (s) 
for  the  exception  and  appropriate  solution  for  its  correc¬ 
tion.  Reports  should  be  simple  and  consistent  with  the 
computer  facility's  responsibility  to  it's  users  in  the 
areas  of  turnaround  time,  reliability,  and  user  support;  see 
figure  3.2  for  a  turnaround  time  report,  figure  3.3  for  a 
reliability  report,  and  figure  3.4  for  user  support  report 
[Ref.  3]. 


Determining  control  limit  values  is  dependent  upon 
configuration  and  capabilities  of  each  individual  computer 
facility,  as  well  as  on  the  goals  of  the  organization.  Past 
performance  may  be  used  with  caution  as  a  standard  against 
which  current  performance  is  evaluated.  Past  performance 
does  not  necessarily  indicate  good  or  poor  performance, 
although  it  is  a  reliable  indicator  of  the  baseline  system's 
natural  reaction  to  various  workload  demands. 

The  perfomance  reports  generated  by  the  CPU  group 
should  always  be  made  available  to  and  discussed  with  the 
user  community.  Publication  of  "performance  charts"  in  the 
facility's  newsletter  or  on  the  bulletin  board  is  an  excel¬ 
lent  vehicle  for  accomplishing  this. 

2 .  Management's  Bole 

The  computer  facility  administrator  (management) 
should  play  a  major  role  in  the  implementation  and  long  term 
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System  Reliability  Report 


Average  Length  of 
System  Interruptions 
(minutes) 


Maximum  Length  of 
Any  System 
Interruption 
(minutes) 


-  Upper 

Control  Limit 


12345  67  (Days) 


-  Upper 

Control  Limit  I 


12345  67  (Days) 


Frequency  of  System 
Interruptions 


-  Upper 

Control  Limit 


7  (Days) 


Frequency:  Dally 

Performance  Criteria: 

1.  The  average  length  of  system 
interruptions  shall  be  less 
than  X  minutes. 

2.  No  interruption  shall  last 
longer  than  Y  minutes. 

3.  There  shall  be  no  more  than 
Z  interruptions  per  day. 


Fifara  3.3  Systaa  Baliability  Baport. 
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management  of  a  CPU  program.  Periodic  review  of  th?  CPS 
effort  by  CPS  management  should  be  performed  to  insure  rhat 
the  basic  goals  of  the  CPS  effort  are  still  in  focus. 

a.  Define  Scope  and  objectives 

As  aenticsed  earlier,  the  CPM  management  should 
develop  and  have  a  clear  understanding  and  definition  of  the 
scope  of  its  responsibilities.  Next,  the  proposed  CPS 
prograa  objectives  should  be  established  and  the  information 
requirements  should  be  defined  in  order  to  insure  that  only 
pertinent  data  will  be  collected  during  the  later  phases  of 
the  CPU  program.  Finally  the  nature  of  the  actual  reports 
to  be  produced,  the  frequency  with  which  they  should  be  put 
together,  and  the  performance  criteria  related  to  each 
should  be  determined.  It  is  important  to  mention  again, 
that  the  reports  should  be  clearly  presented  and  contain 
only  meaningful  and  pertinent  information  for  the  readers. 

0 

b.  Determine  Approach 

The  critical  resource  for  a  successful  CPM 
prograa  is  net  modern,  up-to-date  measurement  equipment,  but 
skilled,  knowledgable  personnel  who  are  intimately  familiar 
with  the  computer  resources  being  measured  and  the  tools 
being  used,  and  who  have  the  analytical  ability  to  appraise 
and  interpret  the  measurement  data.  As  mentioned  earlier, 
without  such  a  resource  and  a  well  conceived  CPH  program, 
the  return  on  the  investment  in  time  will  most  likely  be 
disappointing  if  not  disastrous.  Two  possible  sources  exist 
for  these  key  people:  current  (or  future)  employees,  or 
consultants  from  outside  firms. 

The  former  of  the  two  sources  is  preferred  since 
it  allows  for  better  project  control,  and  has  the  additional 
advantage  that  the  in-house  personnel  are  aore  aware  of  the 
objectives  and  internal  workings  of  the  organization. 
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Some  advantages  of  the  outside  consultant 
approach  are  its  cost-effectiveness  (for  organizations  with 
no  available  performance  measurement  resources--!. e. , 

personnel,  aonitors,  etc.) ,  its  objectivity,  and  the  oppor¬ 
tunity  for  knowledge  transfer  fron  the  consultants  to 
in-house  contacts.  Possible  disadvantages  of  contracting 
out  include  the  consultants'  lack  of  familiarity  with  the 
internal  operations  of  the  organization,  and  the  possible 
friction  that  nay  result  between  the  consultants  and 

in-house  personnel. 

c.  control  and  Review 

"The  failure  of  many  performance  improvement 
efforts  has  been  traced  to  the  lack  of  genuine  management 
interest  and  support"  (Ref.  «]•  A  probable  cause  of  this  is 
having  a  CPH  group  that  is  not  totally  dedicated  tc  the  CPS 
effort,  in  that  CPM  is  not  their  prime  responsibility.  They 
become  spread  "too  thin"  and  if  CPM  has  a  low  priority,  it 
suffers  the  consequences.  Again,  this  problem  can  be  solved 
by  making  sure  that  at  least  one  member  in  the  CPH  manage¬ 
ment  group  has  CPH  as  a  top  priority  and  this  member  can 

dedicate  full  time  tc  the  CPM  program.  Another  possible 

cause  in  the  decline  in  interest  in  the  CPH  program  can 
result  from  false  hopes  and  unrealistic  expectations  for 
cost  savings,  especially  if  major  expensive  measurement 
resources  like  hardware  aonitors  must  be  purchased.  The 
initial  costs  of  a  major  CPH  program  can  be  well  over 
$ 100,000  in  hardware  resources  alone. 

The  total  CPH  program  should  be  reviewed  period¬ 
ically  to  insure  changes  in  informational  need  are  reflected 
in  new  CPH  reports.  Existing  reports  should  be  examined  to 
determine  their  current  relevancy;  irrelevant  reports  are 
often  generated  when  the  need  for  then  has  long  since 
passed. 
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3.  ssssatsa  aiaa&aiaa&i 


Many  of  the  resources  required  for  CPM  have  been 
mentioned  along  with  their  cost.  Perhaps  the  most  signifi¬ 
cant  costs  of  any  CFM  program  are  those  experienced  during 
the  start-up  phase  of  the  effort.  such  costs  usually 
include:  the  program's  initial  planning,  system  modifica¬ 
tions  to  acguire  the  necessary  data,  acquisition  of  commer¬ 
cial  products  and  packages,  and  the  development  of  report 
generation  mechanisms.  Once  underway,  continuous  reporting 
demands  other  costs  and  resources:  system  overhead  to 

collect  performance  data,  processing  time  to  analyze  the 
data,  and  manpower  to  interpret  the  data  and  maintain  the 
reporting  system.  In  addition,  in-depth  studies  prompted  by 
exception  reports  and  an  intolerable  decrease  in  performance 
raguire  the  use  of  rather  sophisticated  tools — notably, 
hardware  and  software  monitors  and  modeling  programs.  The 
cost  of  thesa  tools  varies  with  the  type  and  accuracy  of 
data  to  be  obtained.  The  additional  costs  of  personnel  and 
training  may,  once  again,  make  it  more  cost-effective  to 
hire  outside  consultants. 
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iv.  ££g  tools  M2  122  sisjEg  life  cycle 

As  mentioned  earlier,  the  two  aost  important  and  effec¬ 
tive  tools  for  evaluation  are  visual  inspection  and  common 
sense.  The  use  of  other  tools  without  these  will  most 
likely  be  ineffectual.  The  additional  CPE  tools  available 
include: 

•  Accounting  Data  Programs 

•  Program  Optimizers 

•  Software  Monitors 

•  Hardware  Monitors 

•  Benchmarks 

•  Modeling 

CPE  tools  generally  fall  into  one  of  two  broad  catego¬ 
ries:  measurement  or  predictive.  Accounting  packages,  soft¬ 
ware  and  hardware  monitors  and  source  program  optimizers 
fall  into  the  measurement  group.  Modeling  is  a  prediction 
tool  and  benchmarks  are  a  point  of  reference  for  measure¬ 
ments  and  don’t  really  fit  totally  in  one  or  the  other  of 
these  categories.  These  tools  and  their  use  will  be 
described  as  related  to  the  life  cycle  of  a  computer  system. 

A.  CPE  TOOLS 

1.  Accounting  2lll  SllBSUPB  ££221111 

The  need  for  an  accounting  program  grew  out  of  the 
requirement  to  fairly  distribute  the  cost  of  the  services 
provided  by  a  computer  system  amongst  it*s  users.  Today, 
aost  systems  come  with  accounting  programs  having  varying 
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degrees  of  capability  to  gather  information  about  the  system 
and  individual  prograas.  In  early  multi  programming  environ¬ 
ments,  it  was  noticed  that  essentially  identical  runs  could 
generate  a  wide  range  of  charges.  This  caused  managers  and 
users  to  search  the  details  of  the  accounting  data  collected 
and  to  develop  ways  to  collect  more  detailed  information  in 
order  to  uncover  the  reasons  foe  the  charge  variations 
£Bef.  1]. 

System  accounting  packages  generally  collect  three 
types  of  information:  identity,  quanity,  and  time. 

Identity  data  includes  such  things  as  program  name,  charge 
code,  device  number  and  type,  and  other  information  that  is 
useful  for  categorizing  accounting  data  like  programmer  name 
or  user  name,  type  of  run  (test  or  production),  priority  of 
run,  etc.  Quantity  data  involves  the  amount  of  something 
used  by  a  resource  such  as  the  amount  of  prime  memory 
requested  or  used,  total  secondary  storage  space  used,  count 
of  I/O  requests  to  a  disk  or  tape,  number  of  pages  printed, 
etc.  This  data  can  be  at  a  very  datailad  level,  i.e.  the 
actual  number  of  logical  records  and  both  the  block  count 
and  the  number  of  records  in  a  block.  Time  data  can  show 
the  amount  of  time  used  by  a  job  or  user  for  the  different 
resources  of  the  system. 

Accounting  packages  produce  reports  which  show 
processor  time  use,  total  memory  requirements  of  the  work¬ 
load,  and  give  all  the  statistics  collected  in  a  summary 
format  for  review.  Nodif ications  to  the  accounting  package 
prograas  and  the  development  of  new  programs  can  provide 
user/ job  utilization  reports. 

Accounting  data  can  point  to  those  prograas  that  use 
the  most  system  time  and  resources.  These  prograas  are 
prise  candidates  for  inertased  systea  performance  through 
improved  program  code  and  resource  utilization.  Norris 
[Bef.  1],  gave  exasples  of  this  in  the  chapter  on  accounting 
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data.  In  one  example,  it  was  discovered  that  a  disk  file, 
used  by  a  program  requiring  a  great  deal  of  I/O  time,  used 
unblocked  card  image  records.  'When  the  records  were  blocked 
to  track  buffer  size,  the  program  executed  twice  as  fast  and 
used  considerably  less  secondary  storage.  Another  example 
involves  a  program  using  aultiple  files  that  were  stored  on 
the  saae  disk.  By  placing  certain  files  on  different  disks, 
device  and  channel  contention  was  reduced  which  resulted  in 
increased  effective  utilization  of  the  system  resources. 
This  is  a  case  where  the  hardware  may  have  been  performing 
efficiently  but  the  systea  resources  were  not  being  used 
effectively. 

Accounting  data  can  also  be  used  for  effective  job 
scheduling.  Through  detailed  studies  of  resource  usage  of 
■ajor  jots  a  aore  optimum  job  schedule  'mix’  can  be  achieved 
that  can  increase  system  performance,  especially  in  an 
active  multiprogramming  enviornment. 

Accounting  packages  are  the  most  widely  used  CPE 
tool  throughout  the  industry.  Development  of  more  compre¬ 
hensive  accounting  programs  has  lead  to  more  sophisticated 
and  diversified  applications  of  the  accounting  data  in  terms 
of  system  performance  evaluation.  Morris  [Ref.  1],  says 
that  "One  of  th9  most  promising  developments  of  the  CPE 
field  that  has  largely  been  made  practical  by  the  avail¬ 
ability  of  acconting  data  and  suitable  reduction  packages  is 
Kolences*  Theory  o£  Software  Physics"  [Ref.  10].  According 
to  Morris,  this  work  provides  a  completely  rigorous  defini¬ 
tion  of  software  "work"  and  "power"  in  a  modern  computing 
equipment  context  and  applies  the  physics-like  terms  and 
definitions  to  practical  applications  in  billing  and 
charging  for  system  usage  as  well  as  in  the  aore  encom¬ 
passing  field  of  computer  systea  capacity  management.  The 
text  by  Kolence  is  suited  to  individuals  with  an  interest  in 
the  full  potential  of  accounting  data  and  is  one  possible 
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theoretical  basis  for  an  all-encompassing  description  of 
computer  system  performance. 

2 •  §of tv&£e  flaaitors 

Software  monitors  are  simular  to  accounting  programs 
but  collect  much  more  detailed  information.  They  can  make 
measurements  at  the  step-by-step  instruction  execution 
level.  Software  monitors  are  programs  that  are  attached  to 
the  operating  system  and  capture  event  and  timing  data 
directly  from  internal  tables  setup  and  maintained  by  the 
operating  system.  Data  sample  rate  can  be  varied  by  the 
user  and  captured  data  is  stored  on  tape  or  in  disk  files. 

Software  monitors  are  operating  system/hardware 
dependent.  Even  a  mcdest  operating  system  change  can  impact 
a  software  monitor  program.  Software  monitors  usually  fall 
into  cne  of  three  groups:  sampling,  time-driven  or  event- 

driven. 

Sampling  monitors  extract  a  "sample"  or  subset  of 
the  entire  population  of  interest  and  through  the  use  of 
statistical  methods  inferences  are  made  about  the  entire 
population.  Variation  of  the  sample  size  provides  a  range 
of  confidence  intervals  to  suit  the  accuracy  requirement  for 
a  given  measurement.  Event-driven  software  monitors  rely 
upon  "hooks"  which  are  inserted  into  the  code  to  be  moni¬ 
tored  or  by  a  detection  of  a  change  of  state.  h  hook  is 
simply  an  embeded  instruction  that  triggers  the  collection 
of  predetermined  data  by  the  software  monitor.  Data  collec¬ 
tion  can  also  be  triggered  upon  the  completion  of  "n"  occu¬ 
rences  of  a  hook  if  needed  in  order  to  prevent  excessive 
monitor  overhead  or  "artifact"  from  dominating  the  program 
being  measured.  k  change  of  state  is  defined  as  the  tran¬ 
sition  from  one  activity  to  another  activity.  in  activity 
can  be  a  change  from  privledged  to  unprivledged  use  of  the 
CPU,  the  change  from  CPU  wait  to  CPO  busy,  etc.  changes  of 
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states  generally  occur  less  frequently  then  do  hooks  and 
therefore  performance  data  is  usually  collected  a-  -sack 
occurence.  Time-driven  data  collection  uses  a  clock  or 
timer  and  at  fixed  intervals  of  time  collects  measurement 
data  by  interrupting  the  execution  of  the  program  under 
measurement.  The  most  capable  software  monitors  use  a 
combination  of  these  three  types  of  data  collection  and 
selection  methods.  Software  monitors  use  system  resources 
(i.e.  CPU  cycles,  memory,  etc.)  that  must  be  shared  with  the 
program  or  programs  that  are  the  subject  of  the  evaluation 
effort  and  can  therefore  " contaminate"  the  analysis  by  its 
very  existence. 

3 .  Program  Optimizers 

This  tool  is  a  subset  of  both  accounting  packages 
and  software  monitors.  Actually,  the  program  optimizer  does 
not  directly  optimize  code,  but  rather  it  points  cut  areas 
in  the  program  that  would  produce  the  most  benefit  from 
programmer  provided  optimization.  Since  program  optimizers 
.are  compiled  together  with  the  program  of  interest,  they  are 
computer  dependent.  Program  optimizers  or  "analyzers”  can 
determine  those  parts  of  a  program  with  the  most  activity. 
These  are  called  the  "critical  part"  of  the  program  ana 
typically  "comprise  from  IX  to  10%  of  all  program  statements 
in  a  job  or  program".  [ Hef .  11]  Unfortunately,  program 
optimizers  are  commercially  available  for  only  two  major 
high-level  languages,  FORTH  AN  and  COBOL  [Hef.  1]. 

Application  software  is  not  the  only  software  of 
intarest  when  considering  factors  that  influence  system 
performance.  System  software  (operating  system,  utilities, 
etc.)  also  use  and  consume  valuable  resources  and  should  not 
be  excluded  from  analysis  consideration.  Control  programs 
analyzers  (CPA)  help  to  analyze  this  class  of  software.  In 
some  operating  system  development,  the  only  software 


guidelines  are  that  it  works.  How  efficiently  and  effec¬ 
tively  it  works  are  of  secondary  importance.  A  later 
section  will  discuss  in  detail  the  improvement  of  system 
performance  through  improved  software  practices  in  the  crit¬ 
ical  section  of  a  program. 

4.  jja£dwa£3  Monitors 

Hardware  monitors  are  external  electronic  devices 
that  are  attached  though  the  use  of  probes,  to  the  various 
internal  connections  cf  a  computer  system  to  sense  and 
record  changes  of  state  in  the  components.  An  intelligent 
hardware  monitors  not  only  have  the  capability  to  probe  the 
hardware  but  can  also  be  connected  by  a  line  cf  communica¬ 
tion  to  the  measured  computer  system.  The  intelligent 
monitor  incorporates  its  own  computer,  complete  with  moni¬ 
toring  software  and  uses  this  line  of  communiction  to  actu¬ 
ally  control  the  monitoring  process  via  coordinated 
communication  between  the  monitor  and  the  measured  computer 
system. 

Although  hardware  monitors  are  becoming  "easier”  to 
use,  they  are  still  the  most  demanding  in  the  area  of  tech¬ 
nical  knowledge  required,  especially  in  the  area  of  system 
architecture  and  implementation  down  to  the  circuit  level 
and  require  the  largest  financial  investment.  in  addition, 
hardware  monitors  can  be  easily  misused  or  incorrectly  used 
and  can  generate  data  at  such  a  detailed  level  and  in  such 
large  amounts  that  inexperienced  users  often  find  themselves 
overwhelmed  with  data  after  only  a  short  period  of  data 
collection.  However,  their  low  level  of  detailed  informa¬ 
tion  gathering  capability,  miniminal  operating  overhead  and 
the  fact  that  they  are  generally  system  independent,  makes 
them  an  attractive  and  desirable  monitoring  tool.  The 
choice  of  selecting  between  a  hardware  or  software  monitor 
will  be  discussed  in  greater  detail  in  a  later  chapter. 
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5 .  Benchm^tr  ks 

Benchmarks  are  programs  that  are  designed  to  specif¬ 
ically  represent  the  workload  or  the  set  of  workloads  that 
exist  or  are  anticipated  to  exist  on  current  or  proposed 
future  computer  systems.  Benchmarks  act  as  a  reference  unit 
of  measure  to  be  compared  with  the  results  produced  by  other 
CPE  tools. 

A  small  set  of  benchmarks  that  have  been  validated 

\ 

as  accurately  representing  the  total  workload  at  a  given 
period  in  the  history  of  the  system  can  be  extremely 
valuable  to  a  CPM  program.  Performance  data  produced  by 
periodic  executions  of  benchmarks  allow  the  CPH  group  to 
evaluate  changes  made  to  the  computer  system.  This  periodic 
comparision  process  provides  data  to  easily  chart  the  posi¬ 
tive  (improved  performance)  and  negative  (degraded  perform¬ 
ance)  impact  made  on  the  system. 

6 .  Modeling 

Modeling  is  a  discipline  in  its  own  right  and  has 
been  widely  used  thoughout  many  fields.  Modeling  makes  use 
of  mathematical  descriptions  to  provide  an  analytic  or  simu¬ 
lated  model  of  the  system  to  be  studied.  Modeling  in  the 
computer  enviornment  allows  the  detailed  examination  of  a 
computer  system  or  set  of  programs  before  they  actually 
exist.  In  this  sense,  models  allow  the  "nature**  of  some¬ 
thing  to  exist  without  the  acutal  existence  of  that  thing. 

The  most  important  aspect  of  modeling  is  that  it 
allows  the  question  "what  if  .  .  .?"  to  be  asked  in  a 
variety  of  different  ways  through  the  use  of  parameters, 
with  the  model  producing  results  for  each  of  these  different 
requests.  Modeling  can  also  provide  the  results  of  applying 
a  proposed  solution  as  suggested  through  the  use  of  other 
CPE  tools,  to  a  problem  without  actually  implementing  the 
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solution  suggesxed.  For  example,  a  model  could  try 
different  configurations  of  memory,  channels  and  disk:  units 
without  ever  installing  the  actual  devices.  This  feature  of 
modeling  is  extremely  powerful,  sxpecially  in  long  range 
CPM/CPE  efforts.  However,  a  model  must  be  rigorously  vali¬ 
dated  and  if  possible  verified  with  existing  sytems. 
Developing  a  model  is  very  time  intensive  and  therefore  very 
expensive.  For  a  sore  detailed  discussion  of  these  tools 
see  Morris  [Bef.  1],  in  which  a  chapter  (3-9)  is  devoted  to 
discussing  each  tool. 

B.  CPU  AMD  COMPUTES  SYSTEM  LIFE  CYCLE 

It  is  highly  desirable  to  develop  a  CPU  program  that 
goes  beyond  the  operation  phase  of  the  computer  life  cycle 
and  encompasses  the  complete  life  cycle  from  procurement 
phase  to  transition  phase.  Although  not  suitable  for  all 
computer  facilities  due  to  costs  or  other  reasons,  the 
following  section  lists  some  ideas  to  consider  during  the 
life  cycle  of  a  computer  system. 

1 .  Procurement  Phase 

This  phase  of  the  life  cycle  can  be  the  most  impor¬ 
tant,  in  that  the  framework  for  the  potential  expansion  of 
the  computer  system  is  decided.  This  decision  obviously  has 
the  greatest  impact  on  all  the  following  phases.  It  is  at 
this  phase  that  the  modeling  tool  is  most  powerful  and  most 
enlightening.  After  arriving  at  a  workload  specification, 
modeling  can  be  used  to  allow  alternative  parameters  to  be 
evaluated  for  such  resources  and  components  as  the  type  of 
peripheral  device  best  suited  to  a  particular  file  structure 
or  configuration  of  memory  that  will  be  needed.  Ranges  of 
relative  speeds  and  capacities  can  be  computed  from  the 
model  that  will  allow  selection  to  begin  on  major  equipment 
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such  as  CPU,  memory  and  supporting  equipment  such  as  tape 
drives,  disks  and  printers.  Thus,  the  understanding  cf  the 
proposed  initial  and  future  anticipated  workload  and  equip¬ 
ment  will  aid  in  the  selection  and  development  of  benchmarks 
of  the  system.  Benchmarks  will  play  a  major  role  in 
preparing  the  requests  for  proposals  (RFP) ,  also  known  as 
procurement  documentation,  for  the  pending  equipment 
selection. 

Vendors,  once  aware  that  a  perspective  customer  has 
developed  a  model  capability  and  has  selected  benchmarks  for 
a  system,  are  more  apt  to  become  competitive  by  making  modi¬ 
fications  and  custom  tailoring  their  proposal  to  meet  the 
customers  needs  more  closely  and  at  the  best  price.  "The 
proposal  review  is  one  of  the  most  important  steps  in  the 
system's  life  cycle  and  in  the  CPE  effort  because  it  is  the 
one  time  when  all  potential  suppliers  are  most  intensely 
interested  in  th8  buyer's  needs"  [Raf.  1], 

kt  a  recent  Computer  Performance  Users  Group  meeting 
an  example  was  given  by  a  speaker,  in  which  inadequate 
attention  was  given  to  matching  the  new  system  with  the 
anticipated  workload  involved  a  government  agency  that 
purchased  a  computer  costing  less  than  $300,000.  Having 
procured  the  new  system,  the  agency  proceeded  to  develop 
several  million  dollars  of  software  to  be  run  on  it.  The 
one  system  soon  proved  to  be  inadequate  to  accomodate  the 

workload,  and  several  more  were  eventually  purchased. 

Difficulties  arose  trying  to  interconnect  these  computers, 
and  after  the  purchase  of  five  identical  systems  and 

connecting  them  together  the  system  still  was  unable  to 

provide  sufficient  service  to  handle  the  workload  at  a 
satisfactory  performance  level.  This  situation  would  be 
much  less  likely  to  have  occured  if  a  CPU  program  had  been 
in  effect  daring  the  procurement  phase.  However,  it  is 
thought  that  many  government  organizations  are  forced  to  buy 
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inadequate  computer  power  as  a  rasulr  of  governmentaT-ion 
computer  procurement  procedures.  Larger  system  procurement 
require  more  paperwork  and  have  longer  procurement  times. 
This  may  force  some  organizations  to  buy  a  smaller,  less 
adequate  system  based  upon  the  amount  of  procurement  paper¬ 
work  required  instead  of  the  ability  of  the  system  to  handle 
the  workload. 

The  selection  or  final  step  in  the  procurement  phase 
relies  less  on  performance  factors  since  by  this  time  only 
proposals  that  should  be  considered  have  provided  positive 
benchmark  results,  and  the  remaining  selection  criteria  is 
price,  delivery  schedule,  vendor  support,  conversion  costs, 
etc. 

2.  ISfiSliatias  jhage 

The  two  most  important  concerns  in  this  phase  are: 
establish  the  system  configuration  that  is  best  tailored  to 
the  workload,  and  verify  that  the  delivered  and  installed 
equipment  does  indeed  perform  as  per  written  specification. 
The  former  concern  means  going  over  any  and  all  workload 
characteristics  again  and  verifying  that  the  equipment  and 
configuration  is  correct.  This  is  the  last  chance  to  make 
sure  everything  is  "right".  At  this  phase  the  buyer  has  the 
upperhand  in  that  the  system  is  not  "signed  off"  yet  and  the 
vendor  will  be  motivated  to  be  most  accomodating  at. 
Modifications  that  can  be  simply  made  at  this  phase  become 
increasing  more  difficult  and  expensive  at  later  phases. 
The  latter  concern,  that  actual  performance  equals  or 
exceeds  advertised  values,  should  be  demonstrated  by  the 
vendor. 

It  is  at  this  phase  that  CPN/CPE  personnel  can  learn 
a  great  deal  about  the  new  systee  directly  from  the  vendor 
installation  team.  The  buyer  should  get  involved,  tactfully 
"buy  lunch",  learn  how  to  use  the  system  diagnostics,  and 
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generally  learn  everything  that  zhey  can  from  the  people 
installing  the  system.  If  the  relationship  between  the 
buyer  and  vendor  is  positive  and  accepted  by  the  vendor,  it 
is  more  lilcely  that  the  vendor  technicians  will  point  out 
both  strong  and  weak  points  in  the  system.  The  CPS 
personnel  can  gain  extremely  valuable  information  as  a 
consequence  of  this  relationship  in  that  detailed  technical 
information  relating  to  performance  evaluation  such  as 
system  table  addresses,  pin  location,  probe  points,  etc.  can 
be  obtained  from  current  as  well  as  potential  future 
contacts. 

3.  Operations  Phase 

It  is  during  the  operations  phase  that  most  people 
consider  some  sort  of  CPM  or  CPE  effort.  As  has  been  shown, 
many  potential  problems  reguardiag  initial  and  long  term 
performance  can  be  avoided  or  solved  prior  to  this  phase 
through  the  use  of  a  CFM  program  and  CPE  tools.  Even  with 
all  of  the  prior  CPS  efforts,  a  dynamic  and  expanding  work¬ 
load  will  require  continuous  attention  from  the  CPS  group. 
Each  new  program  should  be  reviewed  as  to  it's  impact  or.  the 
workload  and  performance  of  the  system.  Accounting  data 
along  with  supporting  performance  data  gathering  programs 
provide  sufficient  information  to  assess  the  impact.  If 
problems  are  uncovered,  more  sophicated  CPE  tools  may  be 
required  to  pin  point  the  appropriate  solutions. 

If  a  model  does  exist  that  closely  resembles  the 
system,  '  it  may  be  able  to  determine  the  benefits  and  advan¬ 
tages  of  new  products  and  options  that  become  avaliable 
throughout  the  life  of  the  system.  Through  the  use  of  a 
responsive  and  cost  effective  CPU  program  and  CPE  efforts, 
the  system's  point  of  saturation  can  be  substantially 
delayed.  Thus  putting  off  a  considerable  expenditure  for  a 
major  upgrade  or  new  system. 
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Cnee  again,  a  system  model  is  very  useful  and 
valuable  in  determining  the  capability  of  the  existing 
system  to  handle  an  expanding  workload  and  to  reveal  what  or 
where  system  changes  and  modifications  will  be  required  in 
order  to  satisfactory  maintain  this  workload.  The  limit  of 
this  process  is  system  saturation,  where  additional  enhance¬ 
ments  to  the  system  are  not  possible  or  not  practical  or  not 
cost  effective.  It  is  at  this  time  that  the  computer  system 
will  move  into  the  transition  phase  of  it's  life  cycle. 

4 .  Transition  Phase 

It  this  point,  the  CPJ!  group  should  have  gained  a 
great  deal  of  experience,  become  expert  with  the  dynamics  of 
the  organization's  workload  growth  characteristics  and 
expert  on  a  system  that  has  managed  to  adequately  process 
the  workload  until  this  time.  Along  with  an  indepth 
personal  knowledge  of  the  installation's  performance 
history,  the  CPU  group  should  be  well  prepared  to  aid  in 
system  transition  once  it  has  been  predicted. 

An  organization  that  has  effected  and  benefited  by  a 
CPU  philsophy  and  program  should  be  in  a  very  good  postion 
to  make  an  orderly  and  intelligent  transition  in  which 
major,  unanticipated  performance  problems  will  be  minimized, 
if  they  occur  at  all. 

The  above  discussion  of  CPS  tools  provided  some  very 
general  guidelines  in  the  use  of  these  tools  during  the  four 
phases  of  the  life  cycle  of  a  computer  system.  However,  two 
CPB  tools  that  seem  to  be  used  most  often  throughout  the 
life  cycle  CPH  effort  are  accounting  information  and  system 
and  workload  modeling.  Due  to  the  low  cost  of  accounting 
packages  it  may  be  desirable  to  implement  the  most  extensive 
and  detailed  one  available.  It  should  be  an  ongoing  effort 
to  become  increasingly  knowledgable  about  the  operating 
system  software  so  that  accounting  programs  can  be 
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supplemented  with  additional  capability  in  order  tc  supply 
all  the  information  that  is  desired  in  the  CPU  process.  The 
source  code  of  the  accounting  system  is  an  invaluable  docu¬ 
ment  in  this  regard,  especially  in  answering  the  "where"  and 
"how"  questions  in  obtaining  the  data. 

As  useful  as  models  seem  to  be,  they  are  extremely 
hard  to  develop  and  validate  to  any  degree  of  accuracy  in 
imitating  the  actual  situation.  However,  it  is  suggested 
that,  as  a  minimum,  a  resource  or  component  diagram  be 
constructed  of  the  system  showing  capacity,  speed,  buffer 
size,  transfer  rate,  etc.  of  each  component,  as  well  as 
lines  of  communications  and/or  channels  and  their  associated 
rates  and  capacities.  Some  simple  calculation  using  these 
values  can  be  very  informative  when  trying  to  match  devices 
and  resources.  The  same  idea  could  be  followed  for  the 
workload  where  all  the  programs  would  be  listed  along  with 
the  resources  and  amount  required  as  well  as  critical  time 
values,  such  as  response  time.  Even  the  simplest  cf  models 
can  be  quite  helpful  for  example,  when  components  (and  asso¬ 
ciated  speed,  capacity,  etc.)  of  a  system  are  listed  in  a 
group  as  well  as  a  description  of  will  be  running  on  the 
system.  This  "paper"  model  can  provide  a  feel  for  potential 
problem  areas  such  as  mismatch  of  the  number  of  channels  for 
a  certain  number  of  devices,  or  that  the  amount  of  memory 
presently  configuared  will  only  allow  a  very  limited  multi¬ 
programming  enviornment  to  exist.  Table  I  shows  the  primary 
and  secondary  tools  required  throughout  the  life  cycle 
process  as  proposed  by  Borris  [Ref.  1]. 

C.  PRACTICAL  ARCH1TBCTUAL  EHHAICEBEHTS 

Svcbodova  [Ref.  12],  suggests  that  the  utility  and 
simplicity  of  the  following  architactual  enhancements  would 
make  them  practical  to  implement. 
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•  System  Xll8£«  The  system  rimer  is  a  standard  feature 
of  many  prsent  computers.  Accuracy  of  timeable  meas¬ 
ures  monitored  by  a  software  monitor  is  limited  by 
the  resolution  of  the  system  timer.  The  system  timer 
can  be  used  in  one  or  both  of  its  modes: 

a)  Data  mode:  The  system  timer  provides  time  stamps 
for  monitored  events  (time  of  day  clock). 

b)  Control  godg:  Interrupts  generated  when  the  timer 
expires  drive  the  sampling  routine  of  a  software 
monitor,  or  synchronize  a  hardware  monitor  with 
software  activities  of  the  measured  system 
(interval  timer). 

2.  Instruction  counter.  The  instruction  counter1  counts 
executed  instructions,  either  totally  or  condition¬ 
ally  (instructions  executed  in  problem  state, 
instructions  executed  in  tha  partition  N,  etc.) . 

a)  Data  node:  The  instruction  counter  facilitates 
measurement  of  parameters  such  as  instruction 
execution  rate,  instructions  executed  per  I/O,  CPU 
utilization. 

b)  Control  node:  Similar  to  timer  interrupts,  the 

instruction  counter  can  generate  interrupts  that 
are  used  for  the  purpose  of  program  sampling. 

3.  Memory  cycle  counter .  The  memory  cycle  counter  counts 

memory  references  rather  than  executed  instructions, 
a)  Data  node:  The  memory  cycle  counter  facilitates 

measurement  of  parameters  such  as  memory  reference 
rate,  memory  references  per  instruction,  memory 
utilization,  memory  references  between  page 
faults. 


*The  term  instruction  is  sonetimes  u$ed  for  a  register 
that  holds  the  address  of  the  next  instruction  to  be 
executed  (program  counter,  instruction  address  register) . 
In  this  context,  we  are  using  the  term  instruction  address 
register. 
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b)  Control  node:  The  memory  cycle  counter  can 

generate  interrupts  that  drive  a  monitor  routine 
sampling  program  for  the  purpose  of  examining  the 
locality  of  reference. 

4.  Address  match  circuit.  Many  computers  have  a  hardware 
debugging  aid,  the  address  match  circuit.  The 
address  to  be  matched  is  loaded  manually  thourgh 
toggle  switches.  A  match  generates  a  pulse  in  the 
CPU  circuitry,  but  this  pulse  is  not  recognized  by 
software,  which  puts  a  constraint  on  the  utility  of 
such  a  feature  as  a  monitoring  aid.  To  become  a 
monitoring  aid,  the  address  match  circuit  has  to  be 
given  interrupt  capabilities.  A  special  instruction 
can  be  provided  to  load  the  address  match  register 
under  program  control;  the  absolute  address  that  is 
to  be  matched  then  does  not  have  to  be  known  explic¬ 
itly.  A  match  pulse  causes  an  interrupt  that  trans¬ 
fers  control  to  a  monitoring  routine.  The  address 

match  circuit  thus  provides  a  remote  hook.  Together 
with  an  associative  memory  of  sufficient  size,  the 
address  match  circuit  enables  system  software  level 
instrumentation  without  any  physical  changes  to  the 
monitored  software.  Such  instrumentation  could  be 
temporary  (for  testing  software  monitor  routines),  or 
permanent  (if  for  some  reason  the  monitored  software 
must  not  be  modified).  In  addition,  match  pulses  are 
potential  synchronizing  signals  for  a  hardware 
monitor . 

5.  Monitor  call  instruction.  The  monitor  call  instruc¬ 
tion  provides  access  to  system  monitor  services. 
Such  a  feature  has  been  implamented  on  the  IBM  S/370. 
The  ‘Monitor  Call*  instruction  (HC)  operates  simi¬ 
larly  to  the  IBM  Supervisor  call  (SVC).  A  special 
ccntrol  register,  called  the  monitor  register. 
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functions  as  a  mask  for  different  monitoring  classes 
in  the  same  way  a  aask  is  used  with  system  inter¬ 
rupts.  A  special  privileged  instruction,  'Set 
Control  Register'  can  change  the  aask.  A  'Monitor 
Call*  instruction  then  designates  one  of  the  sixteen 
aonitoring  classes.  If  the  designated  class  is 
unaasked,  then  execution  of  this  instruction  causes 
an  interrupt  that  is  processed  by  the  operating 
system  software. 

6.  saiflaca  i2£iii°£  elna  iatssSsss-  rhe  function  of 
this  interface  is  to  concentrate  signals  representing 
aost  frequently  measured  hardware  events,  and  stabi¬ 
lize  these  signals  for  the  period  of  event  duration. 
The  interface  is  placed  in  a  location  that  is  easy  to 
reach,  with  terminators  that  provide  an  easy  connec¬ 
tion  for  a  hardware  monitor. 
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V.  CHOOSING  £  MONITOR- HARDWARE  OR  SOFTWARE? 


If  the  decision  has  been  made  that  the  accounting  data 
no  longer  provides  the  level  of  performance  information 
required  by  the  CPU  group,  then  the  next  logical  decision 
would  be  to  choose  between  a  hardware  monitor  or  a  software 
monitor.  The  selection  process  reduces  itself  to  picking 
the  monitor  whose  characteristics  most  closely  satisify  or 
meet  the  user's  requirements.  A  very  informative  article 
which  addresses  the  monitor  selection  issue  is  "Performance 
Monitors-Hardware  or  Software?"  [Ref.  13].  The  following 
provides  some  of  the  main  ideas  regarding  the  question  of 
selecting  a  monitor. 

A.  SOME  GENERAL  CONSIDERATIONS 

There  exist  at  least  three  considerations  common  to  both 
types  cf  monitors: 

•  Cost  effectiveness  -As  a  CPM  program  grows,  the  issue  of 
how  cost  effective  the  next  step  is,  should  always  be 
considered.  Purchasing  either  monitor  type  will  demand 
the  allocation  of  more  resources,  both  financial  and 
personnel.  A  monitor  will  incure  the  following  costs: 
the  monitor  itself,  personnel  (both  time  and  training)  and 
most  likely  the  use  of  the  existing  computer  resources  for 
performance  data  reduction  and  report  generation  and 
special  experiments.  The  initial  cost  of  a  monitor  can 
range  from  several  thousand  dollars  to  well  over  one 
hundred  thousand  dollars.  All  of  these  costs  must  be 
totaled  and  compared  to  the  anticipated  benefits  from 
taking  this  step.  The  net  result  will  determine  if  it  is 
cost  effective. 
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•  Definition  of  objectives  -As  with  the  CPM  pro  gran,  a 
plan  is  needed  for  every  important  CPE  endeavor,  espe¬ 
cially  the  purchase  of  a  new  product  or  resource.  Issues 
involve:  how  the  monitor  will  be  used,  what  impact  it  will 
have  on  the  system,  what  will  be  measured,  etc. 

•  Personnel  capabilities — This  issue  has  been  discussed 
earlier  in  this  paper.  The  important  consideration  hare 
is  that  the  monitor  will  require  properly  trained  and 
qualified  personnel  to  use  it  and  these  personnel  must  be 
able  to  communicate  with  vendor  support  personnel. 

1.  Characterist ics  of  Hardware  and  Software  Mor. j-.ors 

Table  II  presents  a  quick  reference  for  comparing 
and  evaluating  twelve  char actaristics  of  the  two  types  of 
monitors  [Bef.  13].  An  elaboration  of  each  of  these  monitor 
characteristics  follows: 

1.  Cost — Since  a  hardware  monitor  is  a  combination  of 
hardware  (some  have  full  capability  microcomputers) 
and  a  software  package,  it's  initial  cost  can  be 
moderately,  to  considerably  more  than  that  of  a  soft¬ 
ware  monitor.  Operating  costs  of  a  software  monitor 
are  considerably  lcwer  than  those  of  a  hardware 
monitor,  since  its  operation  and  configuration  is 
relatively  inflexible,  and  while  set-up  time  for  a 
hardware  monitor  can  be  quite  extensive  (finding  and 
attaching  prcbes,  configuring  logic  boards  or 
programs,  etc.)  it  is  minimal  or  not  required  (y  the 
user)  for  a  software  monitor.  Pormal  training  in  the 
use  cf  a  hardware  monitor  as  well  as  the  design  of 
experiments  also  adds  to  the  cost  of  using  a  hardware 
monitor  more  sc  for  using  a  software  monitor. 

2.  System  overhead — Hardware  monitors  which  only  use 
probes  add  no  additional  overhead  to  the  system  being 
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Characteristics  of  Hardware  and  Software  Monitors 


monitored.  Hardware  monitors  that  have  communication 
links  with  the  monitored  system  can  impose  seme  over¬ 
head.  Software  monitors  can  inflict  substantial 
overhead  on  the  system  since  they  are  programs  which 
are  run  on  the  monitored  system  and  require  the  use 
of  the  system's  resources  such  as  CPU,  primary  and 
secondary  storage  and  I/O  services.  The  severity  of 
the  overhead,  also  referred  to  as  "art ifact" ,  is 
dependent  upon  the  rate  at  which  sampling  or  meas¬ 
uring  is  being  performed.  ifith  the  exception  of  some 
intelligent  hardware  monitors,  both  monitors  rely 
upon  the  measured  system's  resources  for  data  reduc¬ 
tion  and  report  generation.  Some  real-time  systems 
may  not  be  able  to  tolerate  even  the  slightest  amount 
of  additional  overhead  imposed  by  a  monitor,  and 
would  require  a  probe  attached  hardware  monitor. 

3.  Hardware/operating  system  dependent — Mith  the  excep¬ 
tion  of  some  intelligent  hardware  monitors,  hardware 
monitors  are  essentially  independent  of  both  the 
hardware  and  software  of  the  measured  system.  In 
some  cases  inadequate  or  inaccessable  pin  locations 
can  be  a  very  serious  problem,  preventing  full  use  of 
a  hardware  monitor.  software  monitors  use  system 
specific  information  and  are  therefore  custom 
designed  for  a  particular  hardware/software  configu¬ 
ration  for  a  particular  brand  of  computer.  Software 
monitors  don't  exist  for  all  computers  whereas,  if 
performance  related  probe  points  exist  on  a  system,  a 
hardware  monitor  can  be  hooked  up  to  it. 

4.  Precision— Precision  in  a  hardware  monitor  is  depen¬ 
dent  upon  the  following  monitor  internal  components: 
the  resolution  of  the  clock,  speed  of  it's  counters, 
accumulators  and  other  internal  circuitry. 
Therefore,  it  is  desirable  to  use  a  hardware  monitor 
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with  a  precision  greater  then  that  of  the  system 
being  monitored.  Since  software  monitors  utilize  the 
monitored  system's  internal  clock  and  the  ability  of 
the  operating  system  maintained  measurements,  it's 
percision  is  usually  lower  then  that  of  a  hardware 
monitors  potentical  capability.  Due  to  variation 
caused  by  the  sampling  technique  typically  used  by 
software  monitors,  precison  can  drift.  For  the 
majority  of  measurements,  either  monitor  will  prove 
to  be  satisfactory.  In  those  few  cases  in  which 
extreme  precision  must  be  used,  hardware  monitors  are 
necessary . 

5.  Capture  of  qualitative  inforaation--Software  monitors 
can  more  easily  explore  qualitative  information  such 
as  the  contents  of  main  memory,  queue  contents, 
program  ID's,  and  other  descriptive  variables  stored 
in  tables.  The  ability  of  most  hardware  monitors  to 
explore  qualitative  values  is  restricted  by  the  hard¬ 
ware  connections.  Seme  intelligent  hardware  monitors 
do  permit  some  qualitative  information  to  be  trans¬ 
ferred  to  the  hardware  monitor  via  the  use  of  it's 
communication  link.  However,  the  ability  of  hardware 
monitors  to  collect  qualitative  information  is 
limited  relative  to  that  of  software  monitors. 

6.  Training- -Hardware  monitors,  due  to  their  very  tech¬ 
nical  nature,  require  more  training  which  is  critical 
to  the  degree  of  success  attainable.  Hardware  moni¬ 
tors  must  be  configured  by  the  users  which  requires 
selection,  location  and  varification  of  probe  points, 
wiring  of  logic  plugboards,  etc.  In  as  much  as  the 
hardware  monitor  is  system  independent,  it  is 
extremely  nser/analyst  dependent,  and  the  better 
trained  the  user  the  more  benefit  can  be  gained  from 
the  monitor's  capabilities.  Software  monitors 
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require  relatively  little  training  for  implementation 
and  operation.  Software  monitors  have  a  "fixed"  set 
of  reports  and  are  more  easily  analyzed  then  are 
reports  from  a  hardware  monitor. 

7.  Flexibility-- Measure ment  flexibility  potential  exists 

in  both  monitors.  However,  to  achieve  any  degree  of 
measurement  flexibility  in  a  software  monitor,  modi¬ 
fications  must  be  made  to  the  code  in  the  data 
collection  program,  the  data  reduction  program  and 
the  analysis  and  report  generating  programs.  Short 
of  performing  these  possibly  extensible  modifica¬ 
tions,  the  software  monitor  remains  relatively 

inflexible  to  change  of  the  types  of  measurements. 
On  the  other  hand,  the  hardware  monitor  is  designed 
to  be  very  flexible.  In  general,  anything  can  be 
measured  for  which  a  probe  point  exists,  although 
data  reduction  programs  may  be  limited  to  only  fairly 
standard  measurements  and  reguire  modification. 

8.  Base  pf  use--Compared  to  hardware  monitors,  the  soft¬ 

ware  monitor  is  very  easy  to  use.  Activation  and 
selection  of  desired  reports  is  usually  requested 
though  the  input  of  simple  console  commands. 

Hardware  monitors  can  take  several  days  of  set-up 
planning,  coordination  with  the  system  users  in  the 
area  of  scheduling,  and  can  lead  to  disruption, 
confusion  and  interference  in  the  computer  facility. 
Each  new  measurement  which  uses  different  prcbe 
points  requires  reconfiguration  and  planning. 

9.  Portability — There  are  two  major  'divisions  relating 
to  portability:  amongst  different  brand  name  systems 
and  amongst  same  brand  name  systems.  The  hardware 
monitor  is  extremely  portable  in  either  division. 
All  the  hardware  monitor  depends  upon  is  the  avali- 
ability  of  attachable  probe  points.  Software 
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monitors  are  only  portable  amongst  systems  that  have 
the  same  hardware/ software  configuration.  Since 
software  monitors  are  relatively  inexpensive  compared 
tc  hardware  monitors,  it  may  be  as  cost  effective  to 
buy  a  few  software  monitors  for  dissimilar  systems  as 
it  would  be  tc  buy  one  expensive  hardware  monitor. 
One  advantage  of  multiple  software  monitors  over  a 
single  hardware  monitor  is  that  the  software  monitors 
can  run  simultaneously  and  uninterrupted.  &  hardware 
monitor  can  only  be  active  on  one  system  and  must  be 
totally  reconfigured  when  moved  to  another  system. 
One  other  consideration  is  that  hardware  monitors  are 
much  more  susceptable  to  "handling”  problems  in  that 
they  are  delicate  electron  devices,  and  could  be 
easily  broken. 

10.  Device  monitoring — Hardware  monitors  may  be  attached 
to  any  device  having  a  connector.  This  type  of 
attachment  provides  a  different  ”view”  of  the  device 
then  that  possible  through  the  use  of  a  software 
monitor  in  that  hardware  monitors  ”stand  at  the  door” 
to  review  data  entering  and  leaving  whereas  a  soft¬ 
ware  monitor  can  walk  in  and  freely  explore  the 
contents  of  the  room.  This  is  true  in  the  case  of  a 
device  such  as  a  disk. 

11.  Interrupt  considerations — Software  monitors  can  be 
adversely  affected  by  interrupt  conditions  in  the  CPO 
in  that  they  can  be  locked  out,  or  they  can  take 
advantage  of  the  interrupt  capability  to  stop 

.  processing  and  allow  the  software  monitor  to  explore 
any  part  of  the  system.  Since  most  hardware  monitors 
are  external,  they  are  not  affected  by  interrupts. 

12.  Expected  life — Hardware  monitors  can  be  exected  to 
remain  relatively  useful  for  a  long  pericd  of  time  if 
connection  pins  remain  avaliabla.  Software  monitors 
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are  dependent  upon  a  specific  configuation  and  minor 
changes  to  this  configuration  can  impact  the  useful 
life  expectancy  of  a  software  monitor. 

B.  BAKING  THE  DECISION 

As  mentioned  in  the  beginning  of  this  section,  the 
selection  of  a  monitor  is  primarily  a  question  of  matching 
the  specific  needs  of  the  CPU  group  with  the  capabilities  of 
each  of  the  products  avaliable.  The  preceeding  twelve 
points  should  form  the  basis  for  malting  the  comparison  of 
what  is  avaliable.  The  importance  of  initially  defining 
objectives  can  not  be  over  stressed.  The  selection  process 
is  more  apt  to  be  successful  if  it  is  known  beforehand 
exactly  what  is  to  be  measured,  what  questions  are  to  be 
asked,  and  what  desired  end  result  is  expected.  In  some 
instances,  circumstances  may  exist  that  leave  no  choice  but 
selection  by  default.  A  software  monitor  may  not  exist  for 
a  particular  computer  system,  or  a  real-time  or  highly 
active  multiprogramming  system  may  not  be  able  to  tolerate 
additional  overhead  created  by  a  software  monitor,  so  a 
hardware  monitor  is  the  only  choice  by  default.  in  unique 
applications,  such  as  the  space  shuttle  program,  special 
measurements  may  have  to  be  made  requiring  the  minimum  over¬ 
head  of  a  hardware  monitor.  Since  software  monitors  are  by 
far  the  less  expensive  of  the  two,  organizations  that  have 
limited  budgets  may  have  no  choice  other  than  to  buy  a  soft¬ 
ware  monitor,  shortage,  or  nonexistence,  of  personnel  poss¬ 
essing  the  required  backround  or  general  experience 
necessary  to  be  trained  in  the  use  of  hardware  monitors  may 
also  lead  to  the  same  choice.  The  case  studies  found  in 
Bookman  (Sef.  14],  "Saving  from  Performance  fionitoring" 
[Bmf.  15]  and  Sewald  [fief.  16]  provide  ways  which  ether 
organizations  have  elected  to  choose  a  monitor  and  conduct 
measurement  programs. 
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When  laced  with  a  problem,  most  people  will  either  have 
or  will  soon  have  a  "hunch"  or  an  intuitive  explarinatioa  cr 
hypothesis  as  to  what  is  causing  the  problem,  or  what  it  may 
take  to  make  the  problem  go  away.  In  almost  every  case, 
extensive  remedies  should  not  be  implemented  until  "proven" 
to  be  the  correct  or  at  least  the  best  possible  one.  In  the 
context  of  computer  performance  evaluation,  the  best  method 
for  proving  a  hypothesis  correct  is  through  careful  measure¬ 
ment.  If  an  intuition  leads  to  the  implementation  of  a 
remedy  without  adequate  verification  and  proof,  considerable 
time  and  money  can  be  wasted.  Therefore,  one  must  be 
careful  about  the  use  of  ccmmon  sense  guidance  about  where 
bottlenecks  might  be  in  the  systam  and  what  causes  them, 
since  it  is  very  easy  to  be  mislead  in  such  a  complex 
system. 

This  chapter  will  discuss  some  specific,  possible,  and 
general  fallacies  of  intuition  as  noted  by  Carlson  in  his 
article  "Fallacies  cf  Intuition  in  Performance  Tuning" 

[Ref.  17  J. 

A.  SOME  SPECIFIC  FALLACIES 

1.  Fallacy:  During  peak  load  processing  the  CPU  busy 
increases  significantly. 

This  is  a  carry  over  from  the  old  days  when  operations 
people  would  judge  the  behavior  of  the  CPU  by  the  activity 
of  the  noisy  peripherals.  Actually,  the  peak  load 
processing  can  have  very  little  impact  on  actual  CPU  cycles 
used,  but  can  have  significant  iapact  on  noisy  peripherals. 
Additionally,  in  many  cases  it  was  found  that  the  "CPU  usage 


would  increase  only  2%  to  4X  in  CPU  busy"  [Bef.  17], 
however,  a  peripheral  like  the  printer  would  become  twice  as 
busy.  An  increase  then  in  peripheral  activity  would  give 
the  impression  that  the  whole  system  was  much  more  active. 

Not  surprisingly,  and  without  verification,  many 
computer  facilities  would  seek  to  increase  the  capacity  of 
their  system.  This  normally  resulted  in  a  desire  for  more 
CPU  power  or  additional  memory.  Obviously  in  cases  where 
the  actual  increase  in  CPU  activity  during  peak  workloads  is 
very  small,  modifications  to  the  CPU  would  have  little  or  no 
effect  on  the  overall  performance  of  the  system.  What  is 
needed  in  this  situation  is  a  careful  balancing  of  the 
peripheral  eguipment  based  upon  the  results  of  resource 
utilization  measurements,  and  a  possible  review  of 
accounting  data  to  come  up  with  a  better  job  scheduling 
algorithm. 

2.  Fallacy:  The  reason  for  slow  terminal  response  times 
is  slow  disk  access  times. 

Some  simple  comparisions  of  only  time  and  rate  values 
for  cooperating  devices  and  communications  links  could  lead 
to  the  above  conclusion.  Terminals  are  attached  to  control¬ 
lers  (which  have  fast  internal  speeds)  ,  that  coordinate 
CPU's  that  operate  in  microseconds  or  nanoseconds,  and  disk 
units  which  operate  in  milliseconds.  Therefore,  it  would 
seem  intuitively  sensible  that  the  disks,  which  operate  1000 
times  slower  than  CPU,  are  the  most  probable  cause  for 
delay.  Without  going  any  further  than  the  comparisions  of 
the  above  information,  it  seeas  that  obtaining  faster  access 
secondary  storage  devices  would  improve  response  time. 

Carlson  devised  plan  to  measure  those  activities  that 
influence  response  time,  i  hardware  monitor  was  selected  to 
record  all  major  events  occurring  during  response  time 
delay.  Factor  analysis,  a  statistical  technique,  was  used 
to  indicate  which  variables  might  be  influencing  response 
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tiie  the  most.  The  results  appear  in  table  III  and  show 
that  disk  activity  accounted  for  only  14%  of  the  response 


TABLE  III 

Response  Time  Components 


Activity  During 
Response  Tims 


Terminal  Controller 
Disks  Only  Busy 
Channel  busy 
TP  Supervisor  Program 
LCS  inhibit 
Non-interr uptable 
Logical  OR  of  all  Items 
Unaccounted  for 


Percent  of 
Response  Time 


21% 

14% 

5% 

15% 

18% 

46% 

94% 

6% 


time  delay.  Replacing  the  old  disk  with  one  having  one-half 
the  access  tiae  (twice  as  fast)  would  only  result  in  a  7% 
reduction  in  repsonse  tiae.  If  the  access  tiae  were  to  go 
to  zero,  the  net  result  would  be  only  14%  reduction  in 
access  tiae.  In  this  operating  systea  (IBM  OS)  overhead  CPU 
functions  (i.e.  non-interr uptable  activity)  accounted  for 
alaost  half,  or  46%  of  the  response  tiae  delay. 

This  example  frca  Carlson  [Ref.  17),  points  out  that 
there  are  rarely  one-to-one  relationships  between  different 
devices  in  today*s  coaplex  computer  systems.  The  fact  that 
a  disk  drive  operates  one  thousand  tiaes  slower  than  the  CPU 
has  only  a  saall  iapact  or  ainor  relationship  to  overall 
systea  performance.  The  coaputar  system  is  much  more 
complicated  than  a  one-to-one  additive  relationship. 
Therefore,  in  this  case  it  was  not  so  such  the  performance 
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of  the  equipment  or  what  equipment  was  used,  but  rather  how 
things  were  done-i.e.,  the  system  software.  A  close  look 
at  tuning  of  the  interrupt  and  I/O  handlers  could  bring 
about  much  greater  reductions  in  response  time. 

3.  Fallacy:  Switching  from  single  density  to  double 

density  disks  will  cause  system  delay  due  to  arm 
contention. 

Doubling  the  density  while  keeping  the  arm  mechanism  the 
same  can  be  viewed  as  potentially  doubling  the  demands  on 
the  access  arm.  In  view  of  this,  it  is  easy  to  see  how  the 
above  intuitive  belief  could  be  supporxed.  Further  support 
for  this  fallacy  stems  from  the  fact  that  one  would  expect  a 
fairly  high  number  cf  concurrent  seeks  to  occur  due  to  the 
highly  active  operating  system  that  must  support  a  multipro¬ 
gramming  environment  and  disk  controllers  that  support 
multiple  disk  units.  The  fact  is,  that  going  from  single  to 
double  density  has  a  negligible  impact  on  overall  perform¬ 
ance.  Careful  analysis  of  many  operating  systems  indicate, 
however,  that  only  rarely  are  concurrent  seeks  attempted  on 
any  of  the  packs  on  a  given  controller.  The  following 
example  was  found  in  Carlson  [Bef.  17].  A  hardware  monitor 
was  used  on  disk  drives,  clustered  eight  to  a  controller. 
Pins  were  located  that  were  thought  to  provide  evidence  as 
to  whether  or  not  a  seek  was  in  progress  on  any  of  the 
packs.  The  initial  measurements  seamed  to  reveal  that  seeks 
were  in  progress  100*  of  the  time,  indicating  that  the  disk 
controller  was  extremely  busy.  However,  it  was  soon  found 
out  that  the  pins  being  probed  were  only  indicating  if  power 
was  on  in  the  disk  controller.  This  points  out  the  very 
important  need  to  verify  that  the  pin  does  indeed  provide 
the  desired  measurement  information.  One  very  sinple, 
although  not  conclusive  method  to  validate  the  pin,  would 
have  been  to  push  the  halt  button  on  the  system  and  notice  a 
corresponding  halting  of  disk  activity  indicated  in  the 
collected  data. 
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The  correct  pins  were  located  and  verified  ana  it  was 
found  that  on  a  typical  8- pack  disk  subsystem,  overlapped 
seeks  account  for  less  than  of  the  time.  The  maximum 
seek  overlap  ever  measured  was  only  3%.  The  experiment  was 
conducted  prior  to  and  after  the  change  from  single  to  dual 
density  packs.  The  system  was  stabilized  with  single 
density  drives  and  measurements  were  made  of  CPU  activity, 
channel  activity,  response  time,  etc.  Then  the  most  active 
disk  pack  was  combined  with  the  least  active  disk  pack  on 
one  dual  density  drive.  The  same  measurements  were  made 
again  with  no  measurable  difference  in  behavior.  Sven 
combining  the  two  most  active  packs  on  one  double  density 
pack  revealed  no  perceptable  degradation  or  change  in  system 
activity  as  indicated  by  conducting  the  same  test.  Thus, 
careful  measurements  seem  to  dispute  the  intuitive  idea  of 
increased  access  arm  contention  as  a  result  of  switcning 
from  single  density  to  double  density  disk  packs. 
Therefore,  the  decision  could  be  made  to  gtD  to  double 
density  packs  based  upon  economic  issues,  in  that  dual 
density  disks  may  provide  lower  cost-per-bit  secondary 
storage. 

4.  Fallacy:  Large  core  storage  (LCS)  or  asynchronous 
memeory  is  always  a  good  thing  to  put  on  a  system. 

There  seems  to  be  a  widely  accepted  concept  that 
expanded  memory  can  always  increase  the  potential  of  the 
system  to  do  more  work,  even  though  the  memory  may  be 
asynchronous  with  the  CPU  cycles.  This  is  false  as  can  be 
seen  froe  below  and  has  lead  to  some  very  expensive 
mistakes.  The  severity  of  the  problem  is  related  to  the 
difference  in  cycle  time  of  the  memory  and  the  cycle  time  of 
the  CPU.  A  typical  example  of  this  is  found  in  the  IBM  LCS 
on  a  IBM  360/65.  The  cycle  time  for  the  LCS  is  8  microse¬ 
conds  compared  to  1  microsecond  foe  the  CPU.  Asynchronous 
devices  that  must  work  together,  must  get  in  synchronization 
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This  means  the 


with  each  other  before  anyting  can  be  done, 
faster  device,  usually  the  CPU,  must  wait  for  tie  slower 
device,  in  this  case,  the  meaory.  Cache  memory  helps 
relieve  some  of  the  delay  and  speeds  up  memory  I/O 
processing.  In  order  to  achieve  synchronization,  some  fora 
of  "hand  shaking"  is  necessary  to  allow  each  device  to 
communicate  efficiently  and  correctly  with  another  device 
and  assures  that  both  are  ready  to  communicate.  In  the  IBM 
360/65  an  inhibit  pulse  can  be  easily  monitored  by  a  hard¬ 
ware  monitor.  This  pulse  from  the  LCS  travels  to  the  CPU 
and  stops  the  actual  execution  of  instructions.  However, 
"the  meter  continues  to  run",  the  wait  light  stays  out, 
accounting  time  continues,  but  no  instructions  can  be 
executed.  In  this  system,  the  inhibit  time  can  easily  rise 
to  30%  of  total  elapsed  time.  In  essence,  the  CPU  is  halted 
30*  of  the  time. 

Today's  computer  systems  are  a  collection  of  asychronous 
components  including  memory,  CPU  and  channels.  The  fastest 
of  these  must  always  wait  for  the  slower  one  to  finish. 
Devices  that  are  most  closely  matched  in  cycle  speed  to  the 
CPU  are  more  likely  to  have  less  negative  impact  from  the 
inhibit.  Therefore,  the  intuitive  conclusion  of  adding  more 
meaory  to  improve  performance  is  not  necessarily  true,  and 
once  again,  careful  measurements  should  be  made  to  examine 
the  amount  of  time  spent  in  the  inhibit  state. 

5.  Fallacy:  Economies  of  scale  or  "Grosch's  law"  still 
apply  in  computing. 

This  intuitive  comment  may  have  been  more  popular  prior 
to  the  advent  of  minicomputers.  Today,  this  may  not  be  such 
a  wide  spread  feeling  since  costs  of  hardware  have  dropped 
tremendously  with  the  additional  advantage  of  increased 
capability  (CPU  speed,  amount  of  main  memory,  etc.)  .  Herb 
Grosch,  of  IBM  and  past  president  of  kCM,  inferred  that  the 
capacity  or  power  of  a  computer  increased  as  the  square  of 
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the  cost  of  the  computer.  A  study  referred  to  in  Carlson 
[Ref.  17],  indicated  that  minicomputers  can  be  up  to  300 
times  more  cost  effective  (cost/million  matrix  multiplica¬ 
tion  instructions:  high  of  $5,102  for  IBM  360/30  and  low  of 
$16  for  Data  Seneral  Eclipse  from)  than  the  older  mainframes 
or  maxi  computers. 

6.  Fallacy:  In  virtual  storage  systems  you  can  get  by 
with  less  real  memory  than  before. 

Although  virtual  memory  does  simplify  many  problems 
associated  with  multi-task  systems,  it  has  not  been  clearly 
established  that  less  memory  is  required.  In  a  logical 
sense,  the  user  can  be  lead  to  believe  that  they  need  less 
real  memory  in  a  virtual  system.  Althougn  the  evidence  is 
not  totally  conclusive,  the  conclusion  reached  in  the  August 
and  September  1974  issues  of  ED£  Performance  Review  on  7S 
performance  tend  to  indicate  that  additional  real  memory  is 
actually  required  in  order  to  maintain  the  pre-virtual 

memory  system  performance  level. 

B.  SOflE  POSSIBLE  FALLACIES 

A  list  of  possible  fallacies  include: 

1.  Possible  fallacy:  The  use  of  double  or  multiple  buff¬ 
ering  or  a  multi programmed  computer  is  better  than 
single  buffering. 

According  to  Carlson,  these  doss  not  seem  to  be  a  clear 
one-to-one  relation  between  additional  memory  and  channel 
tie  up  in  terms  of  what's  best  for  the  total  system 

throughput.  If  the  data  flow  is  in  short  bursts  on  a  multi¬ 
processing  system  then  the  system  can  often  switch  with 

relatively  little  overhead  to  other  processing  while  the 
transfer  of  data  is  going  on.  Adequate  data  to  prove  or 
disprove  this  intuition  is  obtainable  through  the  use  of 
benchmark  program  runs  with  various  buffer  sizes,  combined 
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with  careful  monitoring  of  the  total  system  activity  at!  the 
total  job  throughput. 

2.  Possible  Fallacy:  Bore  of  a  primary  resource  (main 

memory,  disks,  tape  drives,  channels,  etc.)  will 
speed  up  a  system. 

Very  often  it  is  assumed  that  "acr  s'*  cf  certain 
resources  will  improve  system  performance.  we  have  seen 
earlier,  this  is  not  necessarily  true  for  asynchronous  main 
memory. 

3.  Possible  Fallacy:  If  no  one  is  complaining,  all  is 
well. 

It  is  rare  that  people  stay  satisfied  about  something 
for  very  long.  At  times,  programmers  and  users  may  "accept" 
things  with  the  attitude  that  its*  the  best  the  system  can 
do,  even  though,  say,  the  response  tine  desired  is  not 
currently  being  supplied  by  the  system.  Users  and  program¬ 
mers  should  be  encouraged  to  freely  provide  and  make  sugges¬ 
tions  and  make  known*  their  concerns  and  comments  about 
system  performance  at  all  times. 

4.  Possible  Fallacy:  .  Assembly  language  is  faster  and 
costs  less  then  high  level  languages  like  COBOL. 

Assembly  language  could  be  faster  (in  execution)  in 
special  cases  and  when  used  in  the  critical  secton  cf  a 
program  or  routine.  In  general  though,  medium  to  large 
programs  written  in  assembly  language  are  not  "faster"  and 
less  expensive  then  a  high  level  language  when  the  following 
considerations  are  involved: 

a)  Ceding  time. 

b)  Human  debug  time. 

c)  Hachine  debug  time. 

d)  Compiler/asseably  time. 

e)  Execution  time;  and  frequency  of  execution. 

f)  Program  maintenance  time  and  cost. 
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C.  GENERAL  FALLACIES 


A  wide  range  of  cases  are  covered  by  the  following 
general  fallacies: 

1.  Fallacies  of  generalization. 

This  fallacy  is  developed  when  using  statistical 
sampling  and  measuring  one  event  then  generalializinge  about 
apparently  related  events  as  well.  One  example  is  the  use 
of  CPO  busy  msasurments  to  make  inferences  about  other  parts 
of  the  system  or  the  system  in  general. 

2.  The  fallacy  of  over-reducing  complexity  to 
simplicity. 

This  is  inversely  related  to  the  previous  possible 
fallacy.  If  a  complex  problem  is  overly  reduced  to  a  simple 
one,  then  the  information  obtained  or  the  solution  found  may 
only  apply  to  the  simple  problem  and  in  no  way  be  relevant 
to  the  complex  problem.  Although  at  times  very  desirable, 
simplification  of  problems  must  be  done  very  carefully  and 
not  be  done  entirely  to  reduce  the  detailed  level  of  data 
measurement  and  collection. 

3.  The  Fallacy  of  composition. 

This  is  the  assumption  that  because  one  item  of  a  group 
has  a  particular  property,  the  whole  group  has  the  same 
property.  This  can  occur,  for  example,  if  only  one  of  many 
terminals  or  disk  units  is  measured  and  it  is  assumed  that 
all  others  in  the  group  behave  in  the  same  way.  Again,  the 
data  collected  can  usually  only  be  used  to  describe  the 
behavior  of  the  specific  device  that  produced  the  data. 

4.  The  Fallacy  of  appeal  to  authority. 

Most  people  are  exposed  to  this  fallacy  in  early  life 
and  live  with  it  throughout  their  lives.  Always  check  out 
the  validity  and  extent  to  which  someone  is  an  "authority”. 
If  a  vendor  or  product  salesmen  makes  a  claim,  or  statement, 
or  suggests  a  solution  to  a  problem,  request  the  grounds 


upon  which  the  statement  was  male.  Check  it  out  by 
contacting  other  technical  people  mar  are  directly  involved 
with  the  product  or  method.  For  example,  a  system 
programmer  may  suggest  that  the  system  needs  more  main 
memory  so  that  programs  may  « Jt  have  to  be  "overlay ed”  in 
order  to  execute.  This  may  help  the  programmers  or  ease 
their  problem,  but  could  have  no,  or  even  a  negative  effect 
on  the  system  in  general.  Appeal  to  authority  is  one  of  the 
"cheapest",  quickest,  and  easiest  methods  ro  solve  a  problem 
if  the  information  given  is  correct,  but  it  can  be  the  most 
dangerous  way,  if  the  information  is  incorrect.  Therefore 
it  could  prove  to  be  the  most  expensive  solution. 

The  next  section  discusses  a  suggested  performance 
imporvement  procedure  that  will  hopefully  help  avoid  the 
previously  mentioned  fallacies  through  the  use  of  hypothesis 
formulation  and  verificticn. 

• 

D.  A  SYSTEM  PERFORM  AHCE  IMPROVEMENT  PROCEDURE 

Bell.  [Ref.  3],  in  contrast  to  figure  2.1  "A  common 
approach  to  performance  efforts”,  suggests  a  seven  phase 
testing  procedure  depicted  in  Figure  6.1  and  listed  below. 

1.  Phase-1:  Understand  the  System.  This  is  part  of  the 
CPU  effort  in  terms  of  management  organization  of  the 
installation,  description  and  characteristics  of  the 
workload  on  the  system,  descriptions  of  the  hardware 
configurations  and  software  programs,  etc. 

2.  Phase-2:  Analyze  Ope rations.  This  phase  also 

involves  CPM  and  collects  more  detailed  information 
in  order  to  assess  where  bottlenecks  say  exist  and 
how  well  the  overall  system  is  "running". 

3.  Phase-3:  Formulate  performance  Improvement 

Hypotheses.  Formulate  specific  performance-oriented 
hypotheses  about  causes  and  solutions  to  problems  and 
bottlenecks  uncovered  in  Phase  2. 


Figure  6.1  '  Recommended  Performance  Improvement  Procedure. 


4.  Phase-4:  Analyze  Probable  Cost-Ef  feet  iveness  2%. 

Improveme nt  Modif ications.  Try  to  establish  if  the 

result  will  justify  the  investment  required  by  the 
effort. 

5.  Phase-5:  Test  Specific  Hypothesis.  This  phase 

usually  involves  the  largest  effort  in  the  CPE 
process.  Figure  6.2  from  Bell  [Ref.  3],  provides  an 
orderly  sequence  of  steps  relevant  to  validate  the 
hypothesis. 

6.  Phase-6:  I mpleaent  Appropriate  Combinations  of 

Modifications .  It  may  be  worth  while  to  consider 
performing  several  modifications  at  once  as  opposed 
to  doing  them  one  at  a  time  in  order  to  save  "down 
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Pigure  6.2  Suggested  Hypothesis  Testing  Procedure. 

time”  or  overhead  generated  by  making  a  single  modi¬ 
fication.  Analyze  what  impact  a  composite  modifica¬ 
tion  may  have  on  the  system  and  its  users.  Hill 
programs  have  to  be  modified  and  recompiled? 

7.  Phase-7 :  Test  Effectiveness  of  a odifi cat  ions -Collect 
data  to  establish  the  effect  the  modification  had  on 
the  performance  of  the  system.  If  appropriate  go  to 
phase-3 . 

Bell  provides  valuable  insite  that  is  still  relevent  and 
applicable  to  day  in  the  following  areas:  workload,  opera¬ 
tional  characteristics,  and  system  characteristics.  In  each 
area  he  lists  several  questions  that  should  be  asked. 
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Bell  lists  and 


In  contrast  to  the  fallacies  above, 
describes  several  hypotheses  relating  to  the  three  basic 
methods  he  proposes  fcr  system  performance  improvement.  Ihe 
three  basic  methods  along  with  soma  of  the  hypotheses  are: 

1 .  Seduce  Workload 

a)  Hypothesis:  k  Free-Good  Approach  Has  Lead  to  Large 

Deeands.  This  basically  means  that  if  use  of  the 
computer  is  free  then  users  will  produce  many 
programs  that  will  take  advantage  of  this  and 
possibly  overload  the  system  with  unnecessary 
workload. 

b)  Hypothesis:  Unconverted  Workload  Has  Caused 

Inefficient  Systee  Use.  System  upgrades  usually 
involve  new  and  more  efficient  ways  to  do  things 
(I/O,  new  language  features  such  as  FOSTHAH  77, 
etc.).  However,  many  programs  are  not  modified  to 
take  advantage  of  new  features  and  continue  old 
inefficient  practices. 

2.  Tuning  the  systen 

a)  Hypothesis:  I/O  Contention  for  a  Specific  Device 
Slows  Processing.  This  can  be  a  result  of  placing 
too  much  data  or  too  many  conflicting  programs  on 
one  disk  pack  or  channel. 

3.  Upgrading  the  Computer  System 

a)  Hypothesis:  The  Computer  System  should  Be 

Upgraded  to  a  Multiprocessor.  Increasing  the  CPU 
power  of  a  machine  may  help  in  the  short  run  by 
adding  another  CPU.  If  other  perpherial  devices 
are  adequate  for  multiple  CPUs,  this  can  be  a  cost 
effective  mcve. 

b)  Hypothesis:  The  Current  Systen  Bust  Be  Replaced 
With  a  Larger  Systen.  This  can  result  from  the 
workload  outgrowing  several  components  of  the 
systen  not  just  the  CPU  as  in  the  previous 
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hypothesis.  Upgrading  within  the  sane  computer 
"family"  helps  to  reduce  conversion  costs  and  may 
be  the  most  cost-effective.  However,  caution  must 
be  taken  and  a  review  of  alternative  machines 
should  be  considered  as  well.  A  trade-off  of  easy 
conversion  for  a  modest  in  stead  of  major  increase 
in  UPU  speed  may  not  be  justified. 

B.  SOME  POSITIVE  GUIDELINES 

The  following  axioms  found  in  [Ref.  17],  were  adapted  from 
Fisher's  "Historical  Fallacies"  [Ref.  18]. 

1.  Questions  relating  to  performance  issues  must  be 

operational.  That  is,  thay  can  be  answered  with 

factual  evidence  (supportable  by  measurements). 

2.  The  questions  should  be  open-ended  to  allow  explora¬ 
tion,  but  they  should  not  be  so  wide  open  that  there 
is  nc  direction. 

3.  The  questions  should  be  flexible  so  we  can  adapt  them 
to  new  information.  There  should  be  no  early  hard¬ 
ening  of  the  categories. 

4.  The  questions  must  be  analytical.  This  means  they 

can  be  broken  down  into  smaller  parts  so  we  can 

obtain  answers  to  manageable  pieces. 

5.  The  questions  must  be  both  explicit  and  precise  so 
that  others  can  truly  understand  what  we're  trying  to 
get  at  and  can  concur  with  the  answers  found. 

6.  The  questions  must  be  tested  against  actual  operating 
systems. 

Common  sense  is  a  good  tool  to  use.  However,  as  previ¬ 
ously  shown  it  can  easily  lead  to  fallacies  of  intuition.  To 
use  this  tool  alone  would  be  foolish  and  possibly  wasteful. 
Therefore,  common  sense  should  be  used  to  develop  a 
hypothesis,  backed  up  by  precise  conclusive  measurements. 
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With  each  collection  of  performance  data,  a  simple  binary 
choice  is  not  usually  available.  Instead  of  just  one  fork 
in  the  road  there  can  be  an  overwhelming  number  cf  dead-end 
choices-easy  to  get  cn  and  follow,  but  headed  in  the  wrong 
direction.  Performance  evaluation  is  an  extremely  difficult 
task,  and  can  be  a  wasteful  use  of  valuable  resources  if 
done  improperly.  Experience  is  the  besr  compliment  to 
common  sense.  Both  work  together  to  increase  the  ability  of 
the  individual  or  group  envolved  in  system  performance 
improvements.  With  addition  of  maaningful  measurements  to 
validate  or  invalidate  a  hypothesis  or  intuition,  system 
performance  can  be  realized. 
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VII.  I  BPROVING  PERFORMANCE  SITgOOI  MONITORS 


"Improving  the  performance  of  an  application  system 
doesn*t  always  require  hardware  or  software  monitors,  staffs 
of  specialists,  nor  complicated  timing  diagrams.  Some  of 
the  best  results  may  come  from  just  looking  at  the  code.", 
[Ref.  11].  Computer  programs  are  usually  not  written  under 
the  most  ideal  circumstances  by  the  most  ideal  programmers. 
Many  programmers  don't  become  too  concerned  with  the  impact 
the  program  they  axe  writing,  or  more  cor recti y- their 
"style"  of  programming,  will  have  on  the  performance  of  the 
system  upon  which  the  program  will  execute.  This  view  is 
supported  by  subtle  aud  not  so  subtle  evidence  commonly 
found  in  programs.  For  example,  many  programmers  will  need¬ 
lessly  initialize  an  array  to  zero,  even  though  they  will 
completely  fill  the  array  with  data.  A  less  obvious  example 
is  the  choice  of  data  type  (real  vs  integer)  without  reguard 
to  the  number  of  CPU  cycles  required  for  the  alternative 
choices.  Therefore,  even  the  most  efficient  configurations 
of  hardware  and  system  software  can  still  produce  unsatis¬ 
factory  performance  because  the  applications  that  are 
executed  on  it  may  not  make  the  most  effective  use  of  the 
system  resources. 

The  following  circumstances  can  have  direct  and/or  indi¬ 
rect  adverse  effects  on  the  performance  characteristics  of 
an  application  program  or  systems  program  [Ref.  11]  : 

•  Competence  of  the  program  designer. 

•  The  mandate  given  the  designer. 

•  The  time  frame  under  which  the  "program"  was  produced. 
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•  Tbs  reward  system  in  affect  daring  the  writing  of  the 
program. 

•  Conversions  from  one  hardware  or  software  (operating) 
system  tc  another. 

•  The  number  of  modifications  mads  to  the  system. 

•  The  existance  and  enforcement  of  definitive  programing 
standards  and  quality  control. 

The  list  of  circumstances  that  can  effect  program  perform¬ 
ance  can  become  very  long.  Therefore,  the  chances  of 
improving  overall  system  performance  through  the  improvement 
in  individual  program  performance  has  great  potential. 
Jalics  [Bef.  11]  believes  that,  "Sometimes  systems  can  be 
made  to  operate  two,  three,  or  four  times  as  fast  with 
little  effort".  He  states  further,  "the  task  of  performing 
such  measurement,  evaluation,  and  enhancement  is  not  very 
complex  and  does  not  require  special  skills". 

Jaclis  also  feels  that  there  are  two  basic  principles  to 
be  used  in  performance  measurement: 

1.  "Each  system  has  a  small  number  of  critical  parts 
(these  critical  parts  system2  typically  comprise  1* 
tc  10*  of  all  program  statements  in  a  system)." 

2.  "The  performance  of  the  entire  system  depends  to  a 
large  extent  on  the  performance  of  its  critical  parts 
(that  is,  one  can  just  about  ignore  the  non-critical 
part) " 

This  strategy  for  making  the  more  efficient  is  quite  simple 
and  consists  of  finding  all  the  critical  parts  of  a  system, 
and  modifying  the  areas  that  will  have  a  significant  impact 
on  the  positive  performance  of  tha  system.  Jalics  feels 


2In  the  context  of  this  chapter  the  word  "system"  is 
defined  as  a  collection  of  jobs.  Whereas  jobs  are  defined 
as  a  collection  of  one  or  more  programs.  i  job  step  is  one 
program. 
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that  this  same  strategy  car  be  applied  to  the  whole  set  of 
application  programs.  What  needs  to  be  done  therefore,  is 
to  concentrate  on  a  very  small  part  or  the  whole  (less  then 
10*)  system.  Desirable  selection  criteria,  according  to 
Jalics  are  those  few  application  systems  that: 

1.  Are  time-critical,  that  is,  systems  that  require 
results  a  very  short  time  after  input  data  are 
supplied. 

2.  Are  run  frequently  and  consume  considerable  system 
resources,  or 

3.  Consume  extremely  large  amounts  of  computer 
resources,  even  if  they  are  not  run  very  frequently. 

A.  AH  APPROACH  TO  FINDING  CRITICAL  PARTS 

The  best  source  for  the  determination  of  what  jobs 
constitute  the  ciritical  section,  using  the  above  selection 
criteria  is  daily  and  monthly  job  report  summary  reports. 
The  task  then,  is  to  isolate  those  programs  that  make  up  the 
critical  part  of  the  system,  find  those  parts  of  these 
programs  that  are  executed  most  often  and  review  other  char¬ 
acteristics  of  the  program  that  may  contribute  to  it's  inef¬ 
ficiency.  Jalics  provides  some  guidelines  to  help 
accomplish  this. 

1 .  Check  Out  the  Compiler 

Host  programs  written  in  high  level  languages  are 
feed  as  input  into  a  program  called  a  compiler.  Host 
compilers  are  out  of  the  range  of  modification  by  most 
programmers  unless  they  have  had  compiler  design  and  writing 
experience.  However,  programmers  can  at  least  learn  and 
become  aware  of  the  performance  or  efficiency  characteris¬ 
tics  of  the  compiler.  Specifically,  data  types  and  struc¬ 
tures  and  certain  language  features  can  be  investigated  as 
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to  their  degrees  of  efficiency.  "Data  type  implementation 
alone  can  vary  by  a  factor  of  2  to  4  9  (one  can  be  40  tines 
slower  than  another)"  [Bef.  11]. 

A  siaple  aethcd  by  which  to  look  at  compiler  charac¬ 
teristics  would  be  to  write  a  program  that  contains  all  the 
data  types  and  language  features  of  interest.  After 
compiling  the  program,  an  asseably  listing  of  that  program 
can  usually  be  obtained.  This  listing  can  provide  informa¬ 
tion  about  the  number  of  assembly  instructions  executed  per 
high  level  program  statement.  For  library  routine  calls,  a 
machine  instruction  trace  can  be  used,  if  avaliable. 

Experiments  were  run  on  some  of  the  features  of  a 
COBAL  compiler  residing  on  a  tJHIVAC  1100.  Although  the 
experiments  were  run  against  one  specific  language,  it  would 
not  be  difficult  to  apply  and  conduct  similar  experiments  on 
other  languages.  The  following  were  among  the  fifty-four 
tests  made  by  Jalics  : 

1 .  Arithmetic  statements  with  various  combinations  of 
operands — different  data  field  sizes,  various  align¬ 
ments. 

2.  Movement  (I/O)  of  data  fields  with  sources  and  desti¬ 
nations  of  various  sizes  and  alignments. 

3.  IF  conditions  using  various  lengths  of  items,  align¬ 
ments,  and  use  of  condition-names. 

4.  Table  searching  using  loop,  and  search  verbs,  using 
various  structure  tables,  indices,  data  items,  and 
alignments. 

5.  Language  features  that  examine  fields  using  various 
lenght  fields  and  alignments. 

The  following  conlusicns  ware  drawn  fora  the  results  of  the 
experiments: 

1.  Arithmetic  operations  can  differ  in  execution  by  as 
much  as  30  to  40  times  depending  upon  data  item  type. 
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2.  The  most  efficient  data  field  sizes  may  be  a  multiple 
of  bytes  and  or  words.  The  use  of  these  efficient 
field  sizes  may  increase  processing  2  to  3  times 
faster. 

3.  Alignment  plays  a  tremendous  role  in  efficiency  of 

both  arithmetic  and  character  manipulation. 

Desirable  data  fields  are  single  characters  (no 
alignment  required)  ,  three  characters  (half-word 
aligned) ,  multiples  of  six  characters  (fuil-wcrd 
aligned)  . 

4.  Avoid  testing  large  data  fields  whenever  possible. 
The  use  of  condition  names  for  more  complicated 
testing  offers  no  advantages. 

5.  Built  in  search  verbs  are  not  always  more  efficient 

than  a  program  loop  depending  upon  the  index  used. 
Alignment  of  each  table  entry  to  a  word  boundary 

provides  significant  performance  improvement. 

The  input/output  facilities  are  one  of  the  most 
important  efficiency  aspects  of  a  compiler.  Efficient 
record  and  block  sizes  are  a  function  of  the  characteristics 
cf  the  physical  storage  devices,  as  well  as  the  size  of 

blocks  most  effieiently  handled  by  the  operating  system. 

The  size  of  blocks  can  range  from  several  hundred  to  several 
thousand  bytes. 

2.  Efficiency  Measurement 

An  important  first  step  before  any  performance 
enhancent  is  to  get  a  measure  of  current  efficiency.  What 
is  needed  is  a  way  to  relate  tha  amount  of  useful  work 
performed  to  the  computer  resourcas  it  took  to  process  the 
job,  in  order  to  come  up  with  a  measure  of  work  that  is 
easily  understood  by  the  users  of  the  system  and  the  opera¬ 
tions  group.  Examples  of  measure  of  work  may  be  the 
following:  a  schools  registar's  office  may  have  a  program 


that  does  a  certain  amount  of  procsssing  in  relation  ;o  rhe 
number  of  students  enrolled  in  the  current  academic  quarter 
(i.e.  "X"  number  of  student  records  per  system  uni*  or 

time)  ,  or  a  bank  may  do  a  certain  amount  of  processing  for 
each  tranaction  that  day  ("X"  number  of  transactions  per 
system  unit  of  time) 

Having  obtained  a  measure  of  work,  an  Efficiency 
rate  (ER)  in  terms  of  Uni ts-of- work  (UOW)  divided  by 
Computer-Resource-Uaits  (CRU)  can  be  defined,  [Ref.  11].  In 
the  case  of  a  ONI  VAC  1100,  the  job  accounting  system 
provided  a  resource  unit  measure  as  a  weighted  sum  of  CPU 
seconds  and  I/O  seconds  CRU's  can  then  be  thought  of  as 
system-seconds  with  cne  CRO  equal  to  the  complete  use  of  the 
computer  for  one  second.  Then  the  ER  of  the  registar's 
record  keeping  program  may  be  54  student  records  per  CRU, 
and  the  bank's  ER  may  be  310  transactions  per  system-second. 

The  next  step  is  to  compute  the  ER  for  every  run  of 
a  job.  This  makes  it  possible  to  compare  system  efficiency 
before  and  after  performance  enhancements  are  made,  ever, 
though  different  data  is  used  in  each  run. 

3 .  Obtaining  the  Efficiency  Units 

It  is  highly  desirable  to  compute  and  make  available 
the  ER  at  the  end  of  the  execution  of  a  job.  This  informa¬ 
tion  can  be  placed  in  a  file  for  review  by  interested 
personnel  throughout  the  day  and  then  tabulated  into  a  daily 
ER/jofc  report  or  summary. 

If  the  job  consists  of  only  one  program,  calculating 
the  ER  is  straight  forward,  since  the  UOH  and  the  CRU  are 
easily  obtained.  This  task  becomes  somewhat  more  involved 
with  jobs  containing  multiple  programs  in  that  UOH  and  CRU 
data  will  have  to  be  collected  and  summed  at  the  end  of  the 
job. 
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Finding  Critical  Jo  b  Steps 

If  multiple  job  steps  are  anvolved,  computer  usage 
should  be  captured  at  the  end  of  each  job  stap  or  program. 
Two  sources  of  this  information  ara:  the  operating  system 
may  automatically  put  this  information  in  a  log  or  there  may 
exist  job  control  language  commands  or  supervisory  calls 
that  will  make  available  the  CRU  information  about  the  job 
step.  Having  this  information  about  all  programs  that  make 
up  a  job  helps  to  identify  the  critical  job  steps  or 
programs. 

5.  Critical  Job  Step  Selection 

Having  run  the  system  several  times,  and  identified 
the  critical  job  steps,  a  similar  procedure  is  used  to 
locate  the  critical  sections  within  a  job  step  or  program. 
Once  identified,  the  critical  sections  of  a  job  step  should 
be  locked  at  using  a  compilation  listing  of  the  program  with 
a  cross-reference  index  of  all  data  names. 

6 .  Tune  the  Program  and  Compare  the  Results 

The  next  section  gives  specific  information  on 
tuning  a  program.  The  critical  section  should  be  analyzed 
for  inefficient  code  or  practices  and  modifications 
suggested  to  correct  these.  After  having  implemented  all 
modifications  to  the  critical  section  of  a  job,  using  tuning 
techniques  similar  to  those  in  the  next  section,  the  above 
review  process  should  be  started  anew.  Different  areas  may 
now  become  critical  job  steps  as  a  result  of  modifications 
to  the  "old"  crtical  section.  This  process  should  be 
repeated  as  long  as  it  felt  that  the  rusults  justify  the 
efforts,  which  is  a  decision  unique  to  every  situation. 
Guidelines  on  making  this  decision  should  be  incorporated  in 
the  CPM  plan  of  the  facility,  as  well  as  in  the  individual 
CPE  effort. 
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B.  PROGRAM  TO  MI  KG  HINTS 


Program  tuning  is  within  the  capability  of  most  program¬ 
mers  once  they  have  an  idea  of  what  it  is  about.  As 
mentioned  earlier,  there  is  potential  for  signif leant 
program  performance  improvement,  that  in  turn  can  lead  to 
overall  computer  performance  improvement. 

Jalics  [Ref.  11].  provides  List  of  do's  and  lon'ts 
along  with  ideas  related  to  program  tuning  practices.  The 
following  were  obtained  from  this  list: 

1 .  Find  the  intermost  loops  in  the  program.  A  visibly 
well  constructed  program  may  give  some  hint  as  to 
what  code  may  be  executed  most  often.  Since  the 
contents  of  a  data  variable  may  determine  the 
activity  of  a  loop,  programmers  could  write  a  program 
using  hooks  to  identify  those  parts  of  the  job  step 
or  program  that  are  most  active.  These  parts  usually 
involve  loops  or  innerloops, 

2.  Next,  move  all  superfluous  statements  outside  of  the 
critical  section.  Such  statements  include  initial¬ 
izing  constants,  initializing  records  to  zero  or 
blanks  even  though  they  will  be  filled  with  new  data 
each  time,  initializing  whole  arrays  to  zero  when  no1- 
necessary,  etc. 

3.  Try  to  locate  inefficient  uses  of  data  types  and 
convert  them  to  the  most  effiecient  type  within  the 
intended  precision,  etc. 

4.  Investigate  data  field  sizes.  Different  computers 
may  have  different  optimum  size  boundaries  such  as 
quarter  word,  halfwerd  or  full  word.  For  character 
data,  it  may  be  more  efficient  in  terms  of  processing 
overhead,  to  use  storage  fields  that  are  multiples  of 
words  to  avoid  alignment  overhead  associated  with 
partical  word  fields.  The  overhead  associated  with 
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alignment  becomes  much  nora  obvious  when  inside  a 
loop  that  nay  have  several  thousand  iterations. 

5.  in  coaplex  conditional  statements*  test  for  what  is 
nost  probable  first.  In  tha  case  of  an  I?  statenent 
with  aultiple  conditions*  place  those  conditions 
first  that  are  likely  to  decide  the  result  of  the 
compound  condition  without  testing  all  the  condi¬ 
tions.  (then  conditions  are  ORed*  the  condition  nost 
likely  to  be  true  should  be  checked  first*  for 
AIDing*  the  condition  aost  likaly  to  be  false  should 
be  checked  first.  The  "likelihood"  information 
required  in  the  above  conditionals  is  not  .  always 
known,  but  when  it  is  know  it  may  prove  very  worth 
while  to  make  an  effort  to  take  advantage  of  it. 

6.  In  a  situation  involving  a  series  of  consecutive  IF 
statements*  the  IF  statement  most  likely  to  be  true 
should  come  first  then  the  next  most  likely  second* 
etc. 

7.  Avoid  testing  large  data  fields.  For  example*  it  may 
be  necessary  to  load  a  large  data  field  with  all 
zeros  or  blanks  but  dosn'r  directly  check  the  whole 
field  to  see  if  it  is  indeed  all  zeros  or  all  blanks. 
Rather*  use  a  "flag"  (a  small  status  field)  or  set  a 
bit  in  the  field  indicating  the  fields  zero  or 
nonzero  status.  Checking  a  few  bits  over  several 
words  nay  save  considerable  tine.  This  method  should 
be  used  for  any  large  data  structure  when  appro¬ 
priate. 

8.  If  possible  avoid  the  use  of  verbs  that  are  very 
time-consuming  such  as  COBOL's  TRANSFO RH*  INSPECT  and 
BXABINB  whenever  possible.  one  example  inJalics 
(Ref.  11]*  used  EXARXHE  on  a  record  with  932  charac¬ 
ters  which  resulted  in  6*127  instructions  being 
executed. 


96 


9.  Beware  of  language  interfaces  and  subroutine  call 
overhead.  Overhead  associated  with  certain  language 
invoiced  constructs  can  vary  significantly.  in  COBOL 
for  eras pie  the  efficiency  of  the  PERFORM  verb  versus 
the  CALL  verb  should  vary  greately  depending  upon 
whether  internal  subroutines  are  used  or  external 
subroutines  are  used.  Upon  every  subroutine  call, 
soae  coepilers  will  generats  code  that  will  dynaai- 
cally  obtain  acre  eeoory  to  save  the  environeent  of 
the  calling  routine.  Upon  subprograe  termination, 
the  aeaory  is  then  released.  According  to  Jalics 
soae  coepilers  have  little  known  options  that  will 
either  acquire  additional  aeaory  dynamically  or  else 
reuse  it  each  tiae  the  subprogram  is  called.  A 
simple  example  which  can  be  very  inefficient  in  teras 
of  tiae  and  space  involves  calling  a  subroutine  to 
coapute  a  value  based  upon  the  contents  of  a  field  in 
a  record.  Many  consecutive  records  processed  nay 
have  the  sane  contents  in  the  field  and  a  subroutine 
is  called  each  tiae  to  coapute  the  sane  value.  It 
would  be  auch  aore  efficient  to  retain  the  contents 
of  the  field  and  coaputed  value  of  the  last  record 
processed  and  coapare  it  with  the  current  record  and 
only  coapute  a  new  value  by  calling  the  subroutine 
when  they  differ. 

10.  Know  the  options  available  on  the  conpiler.  Soae 
coepilers  have  options  that  provide  several  levels  of 
effiency.  soae  coepilers  have  aeaory  efficient  or 
execute  tiae  efficient  options.  Code  that  executes 
inline  aay  take  up  aore  aeaory  but  execute  faster 
then  code  placed  in  routines  and  called,  which  uses 
less  aeaory.  Many  other  options  are  available  that 
aay  range  check  data  structures  and  subscripts,  place 
unnecessary  code  on  the  outside  of  loops,  etc. 


However,  some  of  these  options  are  intended  to  be 
ased  only  during  the  final  coapile,  and  if  turned  on 
during  prograa  development  can  waste  significant 
amounts  of  coapile  tine.  Therefore ,  constant  use  of 
inappropriate  compiler  options  during  all  phases  of 
prograa  development  may  have  an  adverse  impact  on 
overall  systea  performance. 

11.  Take  a  close  look  at  tablas  or  arrays.  certain 

alignaent  characteristics  lay  make  the  coapiler 

generate  efficient  code  for  arrays  and  tables.  This 
is  true  in  aachines  that  are  word  oriented  and 
operate  auch  aore  efficiently  on  tables  where  each 
entry  is  vord  aligned.  This  is  a  trade  off  between 
tiae  and  space,  since  table  elements  may  have  to  be 
padded  to  take  advantage  of  the  alignaent  efficiency. 

12.  Look  at  the  subscripts  used  for  addressing  tables. 
In  COBOL  for  instance,  subscripting  can  be  aore  effi¬ 
cient  than  INDEXING. 

13.  Look  at  table  search  techniques.  In  certain  cases 
indexing  (ie.  storing  xxx  in  location  xxx)  can  be 
auch  more  efficient  then  simply  searching  a  table 
[Bsf.  11].  The  indexing  method  is  very  direct.  If 
searching  is  required  make  sure  the  most  efficent 
method  is  used.  This  may  depend  upon  the  order  or 
arrangement  of  the  data  within  the  data  structure. 
In  some  cases,  it  may  be  sore  efficient  to  order  the 
data  to  achieve  better  search  efficieny. 

14.  Is  tbs  current  read  requesting  the  ease  data  as  ths 
last  rtad?  It  aay  bs  worthwhile  to  check  the  current 
rend  request  to  sss  if  it  is  requesting  the  sane 
intonation  as  ths  last  read.  Soae  applications  say 
have  a  high  occurence  of  this,  in  which  case  aany 
unnecessary  I/O  transactions  can  be  avoided. 


15.  Beviev  the  processing  technique  of  a  program,  is  it 
tba  aost  efficient?  Random  accass  techniques  allow 
t he  processing  of  records  in  any  order.  However,  as 
aentioned  earlier,  ordering  the  data  in  the  sequence 
in  which  it  will  be  processed  could  cut  down  on  dish 
access  overhead  tine  in  large  files,  since  the  data 
would  hopefully  be  stored  to  provide  consecutive, 
inline  dish  accesses. 

16.  Is  file  organization  optiaal?  Three  coaaonly  known 

file  organizations  are  sequential,  indexed' 

sequential,  and  direct.  The  choice  of  organization 
used  for  a  file  for  a  particular  application  can 
substantially  impact  performance  of  the  whole  system. 

17.  look  at  file  blocking  buffer  size.  The  programmer 
that  is  aware  of  the  size  of  physical  blocks  which 
are  efficiently  processed  by  the  operating  system, 
the  access  method,  and  the  various  storage  devices 
will  be  better  able  to  produce  efficient  I/O. 
Programmers  that  are  unaware  of  'these  are  more  apt  to 
write  inefficient  code  and  negatively  impact  perform* 
ance.  Block  sizes  on  a  disk  usually  correspond  to  a 
sector  and  the  block  size  for  an  operating  system  is 
constrained  by  how  much  memory  was  allocated  for  the 
I/O  buffer. 
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Till.  £JZ2BS1X  klS  HI28J  IiSIg2t2SX  M£  2S£I2£MAJice 


Increasing  end-user  coaputer  power,  aicro  computers, 
networks,  data  base  machines  and  database  aanageaent 
systems,  multiprocessor  chips,  multiprocessor  computers  and 
fifth  generation  computers  are  all  relatively  new  fields  and 
technologies  that  will  most  likely  demand  some  type  of 
performance  evaluation.  This  chapter  will  explore  some  of 
these  technologies  in  terms  of  performance  evaluation.  New 
performance  parameters  will  have  to  be  defined  as  well  as 
tools  and  methods  to  measure  them.  The  research  literature 
available  in  this  area  is  limited. 

A.  END- USER  COBPOTBB  POVBR 

As  a  result  of  the  favorable  economic  trend  in  the 
computer  hardware  industry,  complete  computers  with  respec- 
tible  computer  power  can  reside  on'  less  than  one  half  of  the 
typical  office  desk.  Performance  issues  in  this  area 
concern  the  capacity  and  reliability  of  the  communication 
linkage  between  the  main  computer  system  and  the  user  equip- 
sent;  the  adaptability  of  this  end-user  equipment  to  the 
various  experience  levels  of  the  group  of  users;  and  the 
amount  of  conversion  of  non-standard  equipment  and  software 
required  in  order  to  be  computable  with  the  main  coaputer 
system. 

B.  HICBOCOaPOTBB  PBBPOIBAICE 

It  is  not  long  after  acquisition  that  the  microcomputer 
user  realizes  the  potential  of  sending  or  receiving  infor¬ 
mation  or  data  to  or  from  other  microcomputer  users.  This 
involves  communication  lines  (usually  existing  telephone 
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lines),  sodess  and  compatability-the  ability  of  the 
computers  to  understand  what  each  is  sending  to  the  other. 
If  we  consider  the  interaction  of  the  user  with  the  ■ ic to¬ 
cos  put  er  as  being  part  of  the  overall  system,  then  user- 
sachine  and  user-scftware  interface  "friendliness"  can 
iapact  performance  of  the  system. 

i  •  caimlaallaD 

As  discussed  in  an  earlier  chapter,  the  rate  at 
which  data  or  inforsation  is  passed  through  a  nodes  can 

severely  impact  the  performance  of  the  computer.  Use  of  a 

slow,  300  baud  modem  between  two  systems  can  produce  the 

sane  performance  as  if  poor  terminal  response  time  and 

system  slow  down  due  to  a  heavy  workload.  The  solution  to 
this  problem  is  simple  but  expensive.  The  user  should 
purchase  the  fastest  modem  affordable.  Unfortunately,  the 
price  of  1200  baud  modems  is  considerably  more  than  the 
slower  300  baud  modems,  and  most  nose  microcomputer  users 
have  300  baud  modems. 

2.  gaiBa&mi&i 

The  other  performance  issue  reguarding  microcom¬ 
puters  that  communicate  with  each  other  is  comparability. 
If  two  microcomputers  are  incompatable  and  can't  communicate 
directly  with  each  other,  then  conversion  or  interface 
programs/hardware  are  needed.  This  adds  overhead  and 
reduces  the  performance  of  each  computer  since  time  must  be 
spent  converting  the  data  or  information  to  a  format  which 
is  understandable  by  computer.  A  standard  interface  chip 
placed  in  each  co nnunicati ng  computer  could  be  a  solution 
to  this  probles.  The  issue  of  ineospatability  becomes  a 
night-mare  when  a  large  organization  like  an  insurance 
company  allows  uncontrolled  proliferation  of  microcomputers 
and  then  decides  to  tie  all  departments  together  into  a 
computer  network  within  the  organization. 


3 •  Beer  Interlace 

The  aicrocoepoter  has  exposed  the  coapater  to  sasses 
of  users.  Hany  of  these  users  siaply  can  not  type  and  are 
frustrated  by  the  use  of  keyboards  that  require  some  degree 
of  finger  dexterity.  Hany  interactive  progress  and  packages 
require  typed  responses,  which  result  in  reduced  systea 
perforaance  when  used  by  this  class  of  users.  Therefore, 
alternative  devices  are  required  to  help  iaprove  the  inter¬ 
action  of  the  user  with  the  coaputer.  These  devices  do 
exist  and  are  called  "picks"  and  include  the  aouse,  the 
light  pen,  the  joy  stick  and  the  touch  panel.  Touch  panels 
seen  to  have  the  nest  reliability  since  they  involve  the 
users  finger  or  a  pencil  and  a  grid  systea  that  detects  the 
location  of  the  pointing  device  relative  to  its  position  on 
the  screen.  Therefore,  the  user  siaply  points  to  an  iten  in 
a  aenue  that  provides  the  user  with  a  choice  of  responses. 
Hewer  systeas  use  touch  panels  together  with  icons-words  or 
ideas  represented  by  an  iaage. 

“•  I&S2E&&51ZS  sstims 

Since  so  nany  aicrceoaputers  have  been  sold,  the 
deaand  for  software,  especially  inexpensive  software  is 
great.  This  tends  to  produce  both  application  and  systea 
software  that  aay  not  be  perforaance  conscious.  Cheap  soft¬ 
ware  nay  run  but  it  aay  run  very  slow  and  use  excessive  aain 
aeaory.  Therefore,  software  should  not  be  purchased  by  price 
alone  but  by  its  reputation  if  possible. 

C.  I1X10BIS 

Telecoaauni cat ions  and  networks  are  becoaing  very 
popular  and  should  see  wide  spraad  use  in  the  1980*s. 
Designers  of  these  technologies  are  providing  a  great  verity 
of  choices  in  the  area  of  data  carriers  and  data  tereina- 
torst 
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•  Front-end  processors 

•  FBI  (private  branch  exchange) 

•  flodees 

•  Voice  satellite  links 

Performance  evolution  in  this  area  has  been  hampered  by 
almost  daily  changes  in  communication  technology. 

Network  design,  development  and  testing  tools  cover  a 
broad  range  of  analog,  digital,  and  protocol  analyses.  The 
physical  components  that  these  areas  deal  with  are:  carrier 
lines,  modems,  and  other  forms  of  communication.  Because  of 
the  potential  size  of  networks  (several  thousand  nodes)  , 
only  software  tools  can  cover  the  overall  global  nature  of 
the  network.  These  software  tools  help  network  designers 
plan,  analyze  and  optimize  the  performance  and  cost  of  the 
system.  Analysis  program  software  gathers  message  traffic 
information,  measures  message  response  times,  determines  the 
probability  of  blocked  messages,  and  ultimately  defines  the 
maximum  traffic  volume  the  system  can  carry  within  accep¬ 
table  response  times. 

Test  objectives  of  protocols  include  overall  evaluation 
of  the  software  and  control  sequences  from  one  end  of  the 
system  to  the  other  end  of  the  system.  On  a  character-by¬ 
char  act  er  basis,  timing  signals  and  properly  controlled 
synchronization  ace  reviewed  as  well  as  error  detection  in 
the  following  three  categories:  characters,  coding,  and 
parity. 

i  central  data  base  is  created  and  used  at  the  system 
level  design  of  a  network.  Various  programs,  (using  param¬ 
eter  information  from  the  data  base) ,  model  many  possible 
configurations  and  analyze  them  in  terns  of  their 
performance  and  cost.  The  sheer  complexity  of  data  commun- 
ciation  networks  (some  nay  eventually  have  7000  on-line 
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units)  necessitates  software  aoaitors.  This  softwareis 
supplied  by  the  vendors  of  front-end  processors  (e. g. ,  the 
IBB  3705  conaunicaticns  processor) .  The  progress  analyze 
the  data  traffic  controlled  by  the  front-end  processor  and 
measure  such  inforaation  as  the  nuaber  of  busy  signals  and 
the  nuaber  of  nonres pending  units.  The  advantage  of  vendor 
supplied  prograas  is  that  they  have  are  able  to  access  all 
coaaunication  lines  controlled  by  the  processor.  In 
contrast ,  protocol  analyzers  and  other  test  hardware  examine 
only  one  line  at  a  time. 

Network  performance  can  be  haapered  by  failed  stations 
or  transmission  components.  Failure  analysis  software  exam¬ 
ines  the  avaliable  channels  and  the  expected  number  of 
failed  devices  per  operating  period.  This  is  done  by  calcu¬ 
lating  the  effectiveness  of  the  fault-diagnostic,  fault- 
control,  and  outage-prevention  aeasures  that  are  built  into 
the  system.  Expected  transaission  errors  in  the  connection 
links  themselves  are  studied  and  added  to  the  expectation 
of  overall  system  performance. 

By  examining  the  potential  data  traffic  capabilities  and 
subtracting  the  effects  of  blocked  messages,  faults,  and  so 
forth,  the  designer  can  measure  the  performance  of  the 
system  with  regard  to  thoughput  (bits  or  characters  per 
second),  messages  per  interval,  and  message  response.  Given 
inforaation  on  traffic  variations,  the  system's  sensitivity 
to  volume  can  be  projected  on  an  hourly,  daily,  and  annual 
basis.  If  a  given  configuration  meets  the  raw  thoughput, 
traffic  volume  sensitivity,  and  reliability  factors,  its 
construction  can  be  optimized  to  lover  costs. 

Ill  of  the  system's  performance  measurements  can  not  be 
optimized  simultaneously.  It  is  in  these  instances  that 
performance  analysis  software  proves  its  worth.  The 
prograas  indicate  the  tradeoffs  among  various  components  and 
methods,  which  allows  cost-effective  system  to  be  designed 
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with  a  reasonable  response  time  and  with  minimum  performar.es 
degradation  in  the  face  of  violations  in  traffic,  error 
rate,  or  failure  rate. 

Data  coaaunication  tests  includeanalog  tests  on  the 
carrier  line  and  digital  tests  on  the  interface  between 
aodeas  and  data  terminal  eguipnent  (computers  and  termi¬ 
nals)  .  The  protocol- level  tests  evaluate  data  and  control 
sequences  character  by  character.  See  figure  8. 1  for  a 
graphical  representation  of  the  data  communications  testing 
environment. 


Figure  8.1  Data  Communications  lusting  Environment 


i.  saaaaai£s&i2a  Lias  lasi&saiiaa 


Existing  telephone  lines  are  used  by  most  networks 
due  to  the  low  cost  and  wide  spread  availability.  However, 
these  lines  are  not  designed  for  the  high  speed,  error  free 
transmissions  desired  by  the  network  and  therefore  cause 
problems.  Factors  that  can  affect  the  performance  of  a 
communications  line  are  analog  parameters  covering  band* 
width,  transmission  mode,  and  transmission  problems. 
Therefore,  the  ana  leg  problems  reduces  itself  tc  testing 
individual  lines  and  determining  if  the  line  meets  the 
network  reguirements  and  specifications.  TIMS  (Transmission 
Impariment  sets)  can  provide  comprehensive  tests  and  meas¬ 
urements  of  an  analog  line.  Measurements  attainable  by  this 
device  are  noise,  the  gain  slope,  noise-to-ground  and 
signal-to-noise  ratios,  and  peak-to-average  ratio.  The 
peak-to- average  ratio  is  a  quick  benchmark  measurement  and 
"gagues  the  overall  spreading  or  smearing  in  time  of  a 
signal  transmitted  across  the  communication  channel” 
according  to  Bailey  (Bef.  19]. 

Measurement  and  testing  of  the  carrier  line  is  best 
accomplished  end-to-end  by  transmitting  a  test  signal  at  one 
end  to  a  meter  at  the  other  end  to  receive  and  measure  it. 
This  simple  method  will  indicate  which  end,  transmitting  or 
recieving,  has  a  problem.  If  the  communications  line  is 
found  to  be  reliable,  then  the  problem  must  be  in  either  the 
modem  or  the  DTE  (data  terminal  equipment)  . 

2.  Data  Ferif ication 

Digital  testing  is  required  to  verify  the  data  being 
transmitted  and  received  over  the  lines.  By  sending  a  fixed 
pattern  test  packet  through  the  system  and  using  a  data 
simulator  or  protocol  analyzer  in  place  of  certain  system 
components,  data  transmission  errors  can  be  identified.  If 
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the  data  becomes  suspect,  the  modem  can  be  checked  vita  a 
pair  of  BEHT's  (Bit  Error  Bate  Testers).  One  BEST  sends  a 
data  sequence  to  the  transmitting  modem  and  another  BEST 
analyzes  the  data  as  it  passes  through  the  receiving  modem. 
&  bit  error  ratio  can  be  established  and  is  a  good  measure 
of  overall  link  performance.  A  typical  voice  communications 
link  should  have  a  ratio  not  exceeding  one  error  in  fifty 
thousand  bits,  according  to  Bailey.  Measurements  capability 
of  BEST'S  vary  vith  manufacturer  and  include  parity  check, 
block  error  rates,  carrier  loss,  clock  slips,  and  timing 
distortions.  The  distribution  of  errors  over  a  given  tit 
stream  is  called  block  error  rate.  Throughput  of  a  system 
that  is  capable  of  block-error  detection  recognition  is 
defined  as  the  ratio  of  error-free  blocks  to  total  blocks 
transmitted.  Blocks  that  are  found  to  contain  errors  must 
be  retransmitted.  This  impacts  throughput  rate,  which  in 
turn  impacts  performance  of  the  system.  A  large  bit  error 
rate  in  conjunction  with  a  low  block  error  rate  would 
provide  tetter  performance  than  an  equivalent  bit  error  rate 
distributed  evenly  across  the  data  stream. 

3.  General  Performance  Summary 

Problems  such  as  carrier  loss,  clock  slippage,  and 
timing  distortions  warn  the  designer  that  problems  exist 
within  the  analog  carrier  line.  The  proper  operation  of 
modems  and  carrier  lines  can  be  verified,  with  the  aide  of  a 
good  tit  error  rate  tester,  through  the  use  of  an  algorithm 
that  isolates  problems  to  various  components  of  the  data 
communications  link,  see  figure  3.2  . 

The  next  step  is  testing  the  OTE,  which  uses  data 
simulation,  data  monitoring,  and  protocol  analysis. 
Protocol  analyzers  have  the  same  capabilities  as  data-link 
monitors  and  simulators,  as  well  as  their  own  data  analysis 
functions. 


Hhen  serving  as  a  monitor,  the  protocol  analyzer  is 
attached  to  the  interface  between  the  modem  and  the  DTE, 
which  nay  be  a  CPU  or  a  terminal.  No  overhead  is  incurred 
by  the  operating  equipnent  from  the  data  analyzer  because 
the  analyzer  is  connected  in  parallel.  Once  the  analyzer 
captures  the  transmitted  data,  the  operator  observes  the 
data  and  protocol  characteristics  to  find  errors.  He  then 
either  freezes  the  fraaes  to  catch  the  data  or  operates  at 
slow  speeds. 

Functions  of  the  equipment  at  either  end  of  the 
channel  are  duplicated  by  the  analyzer  when  it  is  in  the 
simulation  mode.  It  tests  the  part  of  the  data  communica- 
tion  link  that  is  dcwn  or  generates  errors  that  test  the 
ability  of  CPUs  or  terminals  no  recover  from  transaissicn  or 
protocol  errors.  The  instrument  can  also  test  new  equipment 
before  it  is  added  to  an  existing  network  or  laboratory 
system. 

4"  £2£J&ElttS2  1  N£two£k 

The  NDS-II  is  an  Ethernet-based  network  under  devel¬ 
opment  by  Intel.  Perforaance  issues  of  the  network  are 
exaiaed  using  hardware  tools.  &  number  of  Intellec  develop¬ 
ment  systeas  are  tied  together  by  a  resource  manager,  which 
handles  disk  storage  and  files,  to  fora  a  resource-sharing 
network  of  developaent  stations,  k  plugin  controller  board, 
that  contains  an  Ethernet  VLSI  chip  set  and  an  8-  or  16-bit 
processor,  connects  each  station  to  the  Ethernet  network 
(Bef .  19]. 

Data  paths  are  analyzed  by  two  different  hardware 
aonitors  in  the  NDS-II  system.  One  aonitor,  a  ncb  Coaten 
Dynaprobe  data  aonitor  and  analyzer,  is  capable  of  moni¬ 
toring  data  lines  of  up  to  10  BHz  with  10-ns  resolution. 
Signal  levels  and  tranctions  are  counted  using  the  instru- 
aent*s  16  count er-tiaer  probes  when  attached  to  a  data 
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coaauncication  channel.  This  aonitor  aaasures  the  tine  each 
I/O  or  disk  channel  spends  in  various  activities. 

The  second  hardware  aonitor  is  a  proprietary 
performance  analysis  tool,  which  neasures  CPU  activiites. 
It  is  soaewhat  siailar  to  an  in-circuit  eaulator,  but  only 
acts  as  a  passive  data  aonitor.  It  is  conposed  of  two 
Multibus  boards  and  a  buffer  board.  The  processor  to  be 
tested  is  reaoved  free  it's  slot.  The  buffer  board  is 
placed  into  the  socket  connectors  of  the  processor  to  be 
tested,  which  is  pluged  into  the  buffer  board.  This  estab¬ 
lishes  the  analysis  tool  in  a  positon  to  aonitor  the  CPU's 
address,  data,  control,  and  status  lines  and  count  the 
nuaber  of  address  hits.  Hhen  a  trigger  is  tripped,  it  can 
record  the  following  infora ation:  reads  and  writes  to  aeacry 
or  I/O,  operational  code  fetches,  and,  when  monitoring  an 
8086,  op  code  execution. 

Intel,  through  the  use  of  both  analyzers,  was  able 
to  pinpoint  bottlenecks  and  to  increase  the  nuaber  of 
supported  users  per  link  froa  8  users  to  16  users. 

0.  DATA  BASE  HAHAGEBEHT  ST STEM  (DBAS)  PEBPOBHAICB 

DBMSs  are  experiencing  a  constant  increase  in  popularity 
and  use.  A  DBMS  is  a  special  purpose,  coaplex  software/ 
hardware  systea,  requiring  dedicated  resources  of  its  own, 
plus  substantial  resources  froa  the  host  coaputer  systea. 
DBMS  performance  can  have  a  significant  iapact  in  overall 
host  coaputer  system  perforaance.  The  design  of  DBMS  bench¬ 
marks  and  the  interpretation  of  aeasureaent  data  is 
presented  in  a  paper  "Experiments  in  Benchaarking  Balational 
Database  Machines"  at  the  Third  International  tforkshop  on 
Database  Machines  (Bef.  20]. 
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kit ken  (Hef.  21]*  describes  a  DBBS  aodel  that 
provides  perforaance  evaluation  and  prediction  infornation 
and  can  be  used  to: 

•  Predict  the  perforaance  of  a  DBAS  for  a  given  data  base 
design  and  OBBS  workload. 

•  Investigate  DBBS  perforaance  ralative  to  changing  work¬ 
load*  and  perfora  DBBS  stress-test  studies. 

•  Evaluate  alternative  logical  and  physical  data  base 
designs  to  achieve  a  reasonable  perforaance-ef f icient  data 
base  design. 

•  Support  OBBS  perforaance  tuning  and  investigate  OBBS 
perforsance  probleas. 

The  aodeling  approach  results  in  a  DBBS  aodel  which 
closely  siaulates  the  operation  of  a  real  DBBS.  ECSS  II 
(Extendable  Coaputer  Systes  Siaulator  II),  a  special  purpose 
language  for  constructing  discrete- a  vent  siaulation  aodels 
of  coaputer  systeas  (and  also  a  super-set  of  SIBSCBIPT)  was 
used  to  iapleaent  the  DBBS  aodel. 

The  aodel  uses  a  siaulated  Data  Base  Control  systea 
(DBCS) *  which  handles  the  user  interface  and  aanages  access 
to  the  data  base,  I  siaulated  data  base  appears  the  saae  as 
the  real  data  base  conceptually*  but  contains  no  real  data. 
There  exists  a  one-to-one  correspondence  between  the  pages 
and  records  in  the  real  and  in  the  siaulated  data  bases. 
DBCS  operations  include  the  following: 

•  Data  Base  Area  Banageaent 

•  Buff e  isaent— Strategy  and  nusber  of  buffers 

•  Data  .)  e  Banageaent— page  overflow 

•  Journaling- -aanages  a  file  of  before/after  iaages  of  DB 
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•  Data  Dictionary  Iccass 

Tha  aodal  uses  a  Data  Definition  Language  (DDL)  to 
describe  a  complete  data  base  schema.  k  Data  Manipulation 
Language  (DHL)  is  available  which  includes  aost  of  the  oper¬ 
ations  provided  by  the  real  DHL  language. 

k  run-unit  application  prograa  is  represented  proce- 
durally  in  the  aodel  and  consists  priaarily  of  DHL  state¬ 
ments  [ Bef.  21).  Since  the  models  DHL  statements  so  closely 
follow  the  actual  DHL  statements,  the  modeling  of  an  appli¬ 
cation  program's  data  base  activity  can  be  relatively  easily 
accomplished. 

2.  jsiai  fifijs  gstiauftflce  aiiam  aasmi 

in  automatic  feature  of  tha  DBMS  model  collects  and 
reports  a  variety  of  DBAS  and  data  base  related  performance 
statistics  without  user  intervention  and  are  summarized 
below: 

For  each  area  of  a  modeled  data  base: 

•  queuing  and  utilization 

•  number  of  pages  read  and  written 

•  page  access  time 

For  each  simulated  run-unit : 

•  total  execution  time 

•  number  pages  requested 

•  number  pages  read  and  written 

•  records  stored 

For  each  physical  file: 

•  number  of  reads  and  writes 
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•  nuaber  of  seeks  and  zero-tine  seeks 

•  saak  tiaa 

•  data  transfer  tiaa 

•  total  access  tiaa 
For  the  DBAS  buffers: 

•  queuing  and  utilization 

•  nuabar  pages  requested 

•  nuaber  of  pages  read 

•  nuaber  aodified  pages  written 
For  the  DBHS: 

•  run-unit  DHL  request  queuing 

•  run-unit  concurrency  level 

The  DBHS  aodel  closely  reseabled  the  actual  DHBS  and  aade  it 
very  easy  to  use  according  to  Hsu.  Validation  of  the  aodel 
plus  the  addition  of  aore  details  are  currently  underway. 
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Ideally,  every  coepater  installation  should  use  Macro 
Coaputer  Perforaance  Evaluation — CPU  and  be  able  zo  deter- 
aine  vben  and  how  to  use  Micro  Coaputer  Pperforaance 
Evaluation — CPE.  Ideally,  CPM  and  CPE  would  work  togerher 
to  optoaize  the  perforaance  of  the  coaputer  systea.  Muchsel 
[Bef.  22],  suggests  tha  ultimate  coaputer  perforaance 
solution: 

" k  good  solution  to  this  problem  is,  in  our  opinion,  a 
prograa  we  would  like  to  call  the  'automatic  operator' 
(&UTOP)  .  It  analyses  systea  behavior  and  controls  paraae- 
ters  to  aeet  the  operational  goals  of  the  coaputer  installs* 
tioa.  It  either  totally  regulates  operation  of  the  systea, 
or  gives  advice  to  a  huaan  operator  whenever  systea  behav- 
iour  deviates  froa  previously  set  desired  values.  systea 
behavior  is  analyzed  by  a  aodified  perforaance  aeasureaent 
prograa.  How  could  one  optiaize  systea  behaviour  by  such  a 
procedure?  The  aethods  are  to  control  response  fiae;  to 
control  CPO  and  I/O  share  of  certain  types  of  jobs,  possibly 
giving  priority  to  scae  of  thea  according  to  installation- 
specific  criteria;  to  control  systea  parameters  according  to 
tine  of  day  and  workload;  to  recognize  bottlenecks  in  the 
systea  (and  eliminate  thea  if  possible);  to  give  warning 
when  systea  behaviour  deteriorates;  and  to  keep  huaans  well 
inforaed,  reporting  systea  utilization  to  huaan  operating 
personnel  and  tuning  group. " 

Ideally,  coaputer  systea  designers  or  architects  will 
take  into  account  problems  of  perforaance  aeasureaents  and 
their  interpretation  when  designing  new  systeas,  and  aanu- 
facturss  of  computers  will  iapliaent  their  ideas  as  options 
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or  standard  eguipaent  in  the  coaputsr  systea  that  they  make. 
Ideally,  software  designers  will  incorporate  ccmputar 
perforaance  aeasuraent  concepts  into  the  operatling  system 
and  related  systea  software.  Ideally,  lore  and  sore 
coaputer  facilities  will  use  a  "preventative"  computer 
perforaance  evaluation  approach  that  will  replace  the  "emer¬ 
gency  roca"  approach  that  aost  unprepared  coaputer  facili¬ 
ties  are  forced  to  use  when  a  severe  perforaance  problea 
arises. 

Realistically,  the  current  trend  of  decreasing  hardware 
cc3t  s  and  increasing  hardware  capability  sees  to  have 
diluted  the  perforaance  concerns  of  aanagesent  in  many  of 
today's  coaputer  facilities.  Thare  aay  be  a  tendency, 
therefore,  for  coaputer  facility  aanageaent  to  rely  solely 
upon  the  "appeal  tc  authority",  usually  the  vendor 
salesman,  for  a  solution  to  parforaanca  problems  instead  of 
getting  involved  in  a  CPH  and  CPE  effort. 

It  is  intended  that  this  guide  provide  soae  reasonable 
middle  ground  between  the  ideal  situation  and  the  realistic 
situation  that  the  world  of  coaputer  perforaance  seeas  to  be 
in  today.  In  conclusion,  tha  following  staps  are  suggested 
when  considering  perforaance  of  a  coaputer  systea: 

•  Develop  a  CPU  plan  initially  setting  sisple  and  realistic 
goals  based  upon  the  experience  of  the  CPE  personnel  and 
include  all  phases  of  the  coaputer  life-cycle. 

•  Implement  the  best  and  aost  suitable  Recounting  package 
affordable. 

•  locate  "critical  saetiona". 

•  tuna  critical  aacticn  software. 

•  Require  sore  sophicated  CPE  tools  only  whan  existing  tools 
are  considered  inadequate  a  ad  the  CPE  personnel  are  capable 
and  exparianced  anough  to  use  thea. 
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•  If  appropriate f  consider  outside  help* 

Regretfully,  tiee  did  not  permit  the  application  and 
experimentation  of  som  of  the  suggested  tuning  hints  given 
in  this  guide. 
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SOURCES  OF  IHFOBBiTIOM 


The  following  list  gives  additional  sources  of  inforaa- 
tion.  EOF  Performance  Review  annually  provides  an  excellent 
annotated  bibliography  of  the  proceeding  years'  performance 
related  literature.  The  naae  of  the  source  is  in  bold  type 
followed  by  address  inforaation. 
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SOP  perforaance  Review 
Applied  Computer  Research 
P  5  Box  9280 
Phoenix,  AZ  85  0  6  8 

IBB  Systeas  Journal 
IBS 

Arnonk,  NY  10504 


on  International  Conference  on 
Coaputer  capacity  Manageaept 
Institute  for  Software  Engineering 
510  Oakaead  Parkway 
Sunnyvale,  CA  94  086 

International  Syeposiue  on 
Coaputer  Performance  Hodeling, 
Heasurenent,  and  Evaluation 

North  Holland 
P  0  BOX  103 
1000  AC  Aasterdan 
The  Netherlands 


Perforaance  Evaluation 
(sane  as  above) 


Perforaance  Evaluation  Review 
ACH/SI6HETRICS3 1 1  Rest  42nd  Street 
New  York,  NY  10036 


Software  Engineering 
IEEE  Coaputer  Scoiety 
10662  Los  Vaqueros  Circle 
Los  Alanitos,  CA  90720 
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EXAMPLES  OF  PBBFOBHAHCE  BEASUBES 


The  following  performance  measures  were  obtained  from 
Svobodova  [ Bef .  12],  and  are  grouped  according  to  quanity  of 
service,  quality  of  work,  component  utilization,  underlying 
factors,  and  workload  characteristics.  The  measure  is  in 
bold  type  followed  by  the  definition. 

1.  Quantity  of  work  executable  by  a  system 

a)  Throughput:  Amount  or  useful  work  peforaed  by  the 
system  per  unit  of  time  (job  processing  capa¬ 
bility). 

b)  CPU  productivity:  Percent  of  time  spend  in  problem 
state  (used  as  a  measure  of  throughput)  . 

c)  Capacity:  Total  work  executable  per  time  unit  with 
a  balanced  workload. 

d)  System  limits:  Number  of  terminals  that  can  be 

supported  cr  number  of  jobs  than  can  be  multipro- 
grammed  without  serious  degradation  of  service. 

e)  Beserve  capability:  Unused  system  capability. 

2.  Quality  of  Service 

a)  Turnaround  tine:  Elapsed  time  between  submitting  a 
job  to  the  system  and  completion  of  ouput. 

b)  Besponse  tine:  Elapsed  time  between  entering  the 
last  character  of  a  request  at  a  terminal  and 
receiving  the  first  character  of  the  response. 
For  one  time  slice  tasks  Besponse  tine  is  the 
elapsed  tine  to  cosplete  for  tasks  that  require 
less  than  one  CPU  tine  slice. 

c)  Quin  factor:  Total  active  tiae  needed  to  execute  a 
job  six  under  nultiprograsning/total  active  tiae 
needed  to  execute  sane  nix  sequentially. 
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d)  Elapsed  Tiss  Multiprogramming  Factor  (ETHF)  : 

Elapsed  tiae  to  coaplete  a  task  under 

aultiprograaaing/elapsed  tiae  to  complete  when 
this  task  is  the  only  or9  active  in  the  system. 

e)  Internal  Delay  Factor  (I DP):  Active  time  reeded  to 
coaplete  a  task  under  aultiprograaaing/active  time 
to  coaplete  when  this  task  is  the  only  one  in  the 
systea. 

f)  External  Delay  Factor  (EOF) :  Task  elapsed  time/ 
task  active  tiae. 

g)  Reliability:  Probability  that  the  system  functions 
correctly  at  any  given  tiae. 

h)  Availability:  Percentage  of  tiae  a  user  can  get 
systea  services. 

i)  Base  of  use:  Tiae  needed  to  prepare  and  debug 
prograas  for  the  systea. 

3.  Coaponent  Utilization 

a)  Resource  utilization:  Percent  of  time  a  resource 
is  in  use. 

b)  Systea  utility:  Weighted  sub  of  utilization  of 

systea  resources. 

c)  Coaponent  overlap  (device  gain)  :  Percent  of  tiae 
operations  of  two  or  aore  hardware  components  are 
overlapped. 

d)  Queue  length:  Average  queue  length. 

e)  Overhead:  Percent  of  CPU  and  channel  tiae  required 
by  the  operating  systea. 

f)  swap  efficiency:  Frequency  of  occurence  of  indi¬ 
vidual  instruction  operation  codes. 

4.  Underlying  factors 

a)  Coaputer  power:  Huaber  of  operations  per  unit  of 
tiae. 

b)  Raw  speed:  Actual  speed  of  coaputer  coaponents. 
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c)  Reaction  tine:  The  tine  that  elapses  from  wher.  ar. 
input  arrives  into  the  systen  until  it  recisves 
its  first  tine  slice. 

d)  Internal  response  tine:  rime  between  entering  the 
last  character  on  a  terninal  and  receiving  first 
CPO  service. 

e)  Prograe  efficiency:  Anount  of  time  spent  executing 
individual  portions  of  a  progran. 

5.  Workload  Characteristics 

a)  User  interaction  cycle:  Tine  between  initiation  of 
successive  tasks  fron  a  single  terninal. 

b)  User  interaction  rate:  Frequency  with  which  a 

single  user  request  systan  services. 

c)  Service  tiae  (coepute  tine)  :  CPU  tine  required  by 
a  single  task  (job)  . 

d)  Service  rate:  Rate  at  which  requests  are  serviced 
by  the  CPU. 

e)  Arrival  rate:  Rate  of  task  arrivals  to  the 

processor  stage. 

f)  Terninal  tine  (user  think-type  tiae):  Tine  a  task 
spends  in  the  inactive  state  waiting  for  a  user 
response. 

g)  User  intensity:  CPU  tiae  requested/user  think-type 
tiae. 

h)  I/O  tiae:  Service  tine  at  an  I/O  processor. 

i)  I/O  request  rate:  Anount  of  I/O  service  required 
by  a  single  task. 

j)  Active  tiae:  CPU  tine  per  task  core  residence. 

k)  Blocked  tiae:  Tiae  a  task  is  incapable  of 

receiving  CPU  service. 

l)  Degree  of  noltiprograaaing:  Nunber  of  siaultane- 

ously  active  batch  users  or  sinultaneously  active 
user  terainals. 
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